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PHYSICS TOPICS FOR GENERAL SCIENCE EDUCATION 


Lukas Richterek 
Palacky University Olomouc, Czech Republic 


Discussing the content of education is a never-ending story. Naturally, with the development of sciences and 
our knowledge, we need to include new topics. Simultaneously, with limited lesson time, some other parts must 
be reduced or omitted. Thus, each generation has to inventory the most relevant topics for science education and 
define criteria for selecting the curriculum of STE(A)M subjects. It is generally better to cover fewer topics thoroughly, 
practising and consolidating basic facts and skills rather than many of them sketchily. At the introductory university 
or college level, we have many excellent and renowned textbooks e. g., (Halliday et al., 2007; Knight, 2017) or freely 
available textbooks and materials from OpenStax (Urone & Hinrichs, 2022), suggesting the selection of topics. One 
of the key questions for the physics education is the “physics for all’, i.e., for lower and higher secondary educa- 
tion. In the Czech Republic, with ongoing revisions of the national curricula, it is a significant question that may be 
relevant also for a larger audience in the international context. The considerations about the content for general 
education gained more attention from the parents and the public during the COVID-19 pandemic when parents 
supporting their children were more intensively confronted with the content of school curricula. 

The content is developing evolutionally, as documented via textbooks, which remain a key source for teach- 
ers. As shown by (Holbrow, 2008), while 19th-century textbooks were more concentrated on descriptive physics 
with demonstration or measuring apparatus, along with a thorough description of their operation (such books 
can serve today as an encyclopedia of sometimes forgotten apparatus), during the 20th century there was a shift 
towards a higher level of abstraction and emphasizing principles and the unity of physics. According to the analy- 
sis (Keenahan & Keenahan, 2023), there also seemed to be a shift towards textbooks serving more effectively as 
resources for self-directed learning by students rather than as an aid to teachers in giving instruction. Besides, the 
authors and publishers try to make them more visually attractive. Both tendencies are undoubtedly even more 
observable in the case of online resources. 

It is a part of the national tradition whether to teach science subjects separately (biology, chemistry, phys- 
ics, etc.) or integrate them into one subject, e.g., science. Both possibilities have their advocates, therefore the 
national context plays an important role. As shown by (Holubova, 2024), from the central European view, physics 
should not be replaced by integrated science, despite the current shortage of physics teachers. Ideally, physics 
as a subject should support building skills considered important in the 21st century, namely communication and 
collaboration skills, creativity, and critical thinking. As briefly summarized by (Muller, 2010), “physics is the liberal 
arts of high-technology" with significant contributions to advances in new technologies that arise from theoreti- 
cal breakthroughs. This was and sometimes still is reflected in the school curricula, that physics may start before 
other subjects like chemistry, which build on some physics concepts (Sheppard & Robbins, 2003). Based on the 
experience with teaching live-sciences majors, (Meredith & Redish, 2013) also point out, that two disciplines like 
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physics and biology see the world differently (though, of course, there are always exceptions and overlays). In 
general, physicists stress reasoning from a few fundamental principles (usually mathematically formulated) and 
seek to build understanding from the simplest possible models paying much attention to constraints, such as 
conservation laws. Biologists deal more with real examples and emphasize structure-function relationships, be- 
ing more descriptive and stressing less quantitative reasoning arising from “fundamental” abstract principles or 
simplified pictures. Biology (and sometimes also chemistry) students are typically less fond of mathematics and 
are reasonably comfortable with using equations to get quantitative results but are less familiar with using them 
as pieces of compressed information and insight into the world. This is even more true about the non-technically 
or art-oriented students. This brings other skills that are left to be developed within physics classes, like drawing 
inferences from equations or relations, connecting equations to physical meaning, understanding the implications 
of scaling and functional dependence, making estimates, and establishing a sense of scale; all this with appropriate 
respect to the level of education. Arguing for the importance of physics, we must also appreciate all other STEM 
subjects necessary for understanding our world’s complexity and for the solution of its urgent problems, e.g., 
environmental (Lamanauskas, 2023; Lamanauskas & Malinauskiené, 2024). 

The Ministry of Education, Youth and Sports of the Czech Republic recently left out e. g. Newton's laws and 
Ohm's law in physics, or the cell in biology from the obligatory national curriculum for lower secondary education 
with the rationale that those topics are difficult and abstract parts that are anyway not understood by most of the 
pupils which leads to the pure memorizing of the facts. Of course, teachers can take these topics, if they want, 
but it is not obligatory. This again opened more intensive debates over the curricular content. Undoubtedly, most 
schoolchildren will not intentionally apply Newton's laws in their lives (not realizing the laws work independently, 
whether we know them or not), and probably even some non-physics scientists cannot recite Newton's laws several 
years after graduating from their high schools. Although Ohm's law is a nice and rare example of an exact linear 
dependency in physics, the situation is probably similar. More pessimistically, most of what we try to pass on to 
pupils will probably be forgotten after some time, and, on the contrary, pupils will, if life leads them to it, acquire 
knowledge of what we did not give them (namely in their hobbies). From that point of view, should “physics for 
all” content be reviewed by non-physicists (e.g., biologists or chemists) and vice-versa? 

The question is, whether the preparation for life is the only task of school. Speaking about general educa- 
tion, isn’t one of the school’s roles to spread out before pupils the panorama of the natural and human world that 
surrounds them? To help them join and appreciate human culture, where sciences, including physics, naturally 
belong? From that point of view, Newton's laws are part of a great revolution in history —- they have shown us how 
rich and variable natural processes are governed by a small number of intelligible principles that form a coherent 
and logical whole that enables us to explain (among others) the mystery of celestial motions and the Earth's grav- 
ity. From that point of view, can we imagine a “history for all” curriculum without mentioning the French Revolu- 
tion or a world literature review without Shakespeare? From this perspective, it provides an opportunity for “the 
power of story” (Willingham, 2021), and for the creative teachers, there are a lot of resources to make it with four 
Cs: Causality, Conflict, Complications, and a strong, interesting Character. Those components are natural parts of 
most great scientific discoveries. 

Thus, we gradually come to the key question, what topics should be included in secondary-level physics as 
“physics for all”? We may look for inspiration in many textbooks intended for non-physicists, as examples from many 
let us mention (Crowell, 2016, Crowell, 2020; Knight, Jones, & Filed, 2022, March. 2002) or (Muller, 2010) intended 
as a physics for future world leaders. In its content we can find the following chapters: 

e Energy and power, the physics of explosions. 

e — Atoms and heat. 

¢ Gravity, force, and space (with subchapters on rockets, airplanes, Newton's 3rd law of Interactions). 
e Nuclei and radioactivity, chain reactions, nuclear reactors, and atomic bombs. 

e — Electricity and magnetism. 

e Waves (UFO's, earthquakes, music, light, invisible light). 

¢ Quantum physics (lasers, transistors, electron microscopes, quantum computers). 

e Elements of special and general relativity. 

e The Universe (solar system, expanding universe, dark energy, and the beginning). 

The problem is that such selection more or less, maybe in non-traditional order, covers more or less most of the 
topics we are traditionally teaching, not leaving many suggestions, what should be omitted. As mentioned below, 
the “devil is in details”— the content of those parts and, as often pointed out, in the way of exposition, used methods 
and activities. By the way, does anybody remember such topics being an important part of the pre-election debates? 


595 


https://doi.org/10.33225/jbse/24.23.594 


Journal of Baltic Science Education, Vol. 23, No. 4, 2024 


PHYSICS TOPICS FOR GENERAL SCIENCE EDUCATION ISSN 1648-3898 | /erint, 


Mee BOS Aaa ISSN 2538-7138 /oniines 


A specific problem is the inclusion of modern physics parts. Though this is a little bit relative term, usually it is 
understood as the physics of the 20th century, namely relativity, quantum and particle physics and the physics of 
microworld (which corresponds to the last three items in the list above). For example, the basis of special relativity 
theory is more than a century old and brings the spontaneous attention of the public, but relativistic effects like 
length contraction, time dilation, etc., are not supported by our everyday experience and intuition. Therefore, it is 
usually considered a difficult topic by most students and by many secondary school teachers (Ozcan, 2017). There 
are many valuable attempts to make this topic more accessible and understandable at various levels of education 
e.g., (Alstein et al., 2021; Kersting & Blair, 2021). In Czech grammar schools, special relativity was included in the 
physics curriculum in the sixties, which was followed by an effort to evaluate the study materials and bring some 
innovations (e.g., omit the Lorentz transformations). Since that time, the content of the relativistic curriculum at most 
grammar schools has been reduced in lesson time, so now it roughly corresponds to a very simplified version of 
chapter 37 (Halliday et al., 2007) with fewer examples and problems. Though it may slightly vary in school curricula, 
the key topics are the same as introduced more than 50 years ago and covered in the most used textbook: space 
and time in classical mechanics, the historical development of SR, basic principles (postulates) of SR, the relativity 
of simultaneity, time dilation, length contraction, relativistic addition of velocities, basics of relativistic dynamics, 
mass-energy relation and Einstein’s biography (another powerful story supported by TV series Genius and various 
biographies, locally we can recall his work at the Prague university in the years 1911-1912). Among others, the 
knowledge gained from studying the fundamentals of relativity is also applicable to teaching the fundamentals of 
quantum physics, atomic physics, and nuclear physics, it enables students to better understand the basic concepts 
of physics (energy, mass, time, simultaneity, etc.), that are not straightforwardly intuitive. The optimistic message 
provided by (Kodisova et al., 2024) is, that the performance of the Czech students in the test on special relativity 
has been nearly unchanged within the decades, despite some reduction of the lesson time. 

Despite its importance, physics is typically unpopular as a subject, namely from the higher secondary level, 
where it is traditionally more based on math. Some authors emphasize the relevance of physics concepts in everyday 
life analogies (Das, 2015). The importance of scientific knowledge for survival in severe conditions was illustrated 
many times, from Verne’s classical novel The Mysterious Island to a space Robinsonade version, Weir's novel and 
the following movie The Martian. Non-formally, these fictions are a unique kind of argument for education as life 
experience (knowledge, attitudes and skills) for life (cognition, consideration, behaviour), the idea formulated 
empathically by (Broks, 2014; Broks, 2020) as one of the top values to be globally developed today. As far as the 
content is concerned, we come to the conclusion that there are no answers for all concrete situations and needs. As 
technical details today are rapidly changing, more attention and value should be attributed to the understanding 
of general principles rather than particular details. This brings another advantage - the principles and laws have 
long-term validity (just remember the Newtonian laws published for the first time in 1687, Einstein's relativity only 
sets limits for their validity). 

In the end, as put by (Brooks, 2014), our general aim, not only in STEM education, is to help our students 
to become creative, responsible, clever and honest people for our lives in the future. So, the content we choose 
should help them to develop general corresponding skills like critical thinking, creativity, communication and col- 
laboration (Abersek, 2023). When preparing future STEM teachers, we can assume that students choosing teacher 
preparation programmes are generally not worse in scientific performance and technical subjects (Hrouzkova & 
Richterek, 2021). On the other hand, some students start to fully develop their basic reasoning abilities around 
their college years and need support even at that phase, especially in tasks demanding hypothetico-deductive 
thinking and correlation thinking. 

JBSE traditionally supports raising the quality of physics education (Demkanin, 2023), which is a never-ending 
adventure. We know that as far back as the late 1800s, U.S. physics teachers expressed many of the same ideas 
about physics education reform that are advocated today; the situation in other sciences is highly probably analogi- 
cal. According to (Holbrow, 2008), the physics teachers of 1899 complained of declining enrollments and lack of 
student motivation. Some asserted the need for hands-on experimentation, some urged that students be actively 
engaged. In every decade, there has been debate over the level of mathematics and mathematical preparation 
appropriate for beginning physics. Despite the widespread and long-standing support for the “inductive” method 
of instruction — referred to by various names such as “inquiry,’“scientific practices,” etc. - we still must support and 
advocate for the successful, long-lasting, and broad-based implementation of this method either in high school or 
college physics courses (Meltzer & Otero, 2015; Otero & Meltzer 2016), finding positive examples of these desired 
methods logistically and practically integrated into normal classroom practices. Besides repositories of shared 
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teaching materials like Compadre Portal (https://www.aapt.org/ComPADRE) maintained by the American Association of 
Physics Teachers, especially the sharing of experiences in teachers’ communities seems to help to gradual steps in 
the desired direction (creating “islands of positive deviation’). From the Czech context, we can mention, e.g., every 
year conference Physics Teachers’ Inventions Fair (https://vnuf.cz/index-en.html) or the Heureka Project (Physics must be 
experienced!) (https://kdf.mff.cuni.cz/heureka), in Lithuania, e.g., National Scientific Practical Conferences organized by 
SMC Scientia Educologica (http://gu.puslapiai.It/conferences_en). ln general, teaching STEM subjects is a complex mixture 
of practical and factual knowledge, conveying a deep understanding of fundamental principles and developing 
the appreciation of, and facility with, the use of scientific methods. To some extent, the teacher can choose the 
right proportions (not leaving any part) and the corresponding content and topics in some consistent and logical 
order appropriate to the level of the students. Naturally, the teachers should be comfortable with the topics, ide- 
ally positively interested in them; nobody can expect that the teacher is going to raise the interest of students if 
he or she does not like it and has no basic interest in it. Therefore, we should always keep some disposable lesson 
time for the teachers to go through the content they know and enjoy. Then we can meet the case that effective 
teachers can connect personally with students and organize the material in a way that makes it interesting and 
easy to understand (Willingham, 2021). 

This year we have lost Daniel Kahneman (March 5, 1934 - March 27, 2024), an Israeli-American psychologist 
best known for his work on the psychology of judgment and decision-making as well as behavioural economics, 
for which he was awarded the 2002 Nobel Memorial Prize in Economic Sciences together with Vernon L. Smith. In 
the educational context, he is known for his bestsellers (Kahneman, 2011; Kahneman, Sibony & Sunstein, 2021). 
According to him, our thinking consists of 2 systems. System 1 is fast, automatic, and intuitive, operating with little 
to no effort. This mode of thinking allows us to make quick decisions and judgments based on patterns and expe- 
riences. In contrast, System 2 is slow, deliberate, and conscious, requiring intentional effort. This type of thinking 
is used for complex problem-solving and analytical tasks where more thought and consideration are necessary. 
More often than we would like to admit, we rely on System 1, which can generate surprisingly complex patterns 
of ideas, but only the slower System 2 can construct thoughts in an orderly series of steps, consider probability, 
etc. From this point of view, within STEM education our goal is clear — to support System 2 and the habit of using 
it more often. Alternatively, as put by (Willingham, 2021), we should review each content and lesson plan in terms 
of what the student is likely to think about. As memory is the residue of thought, we must ensure that students are 
thinking about the meaning of the material and content of our lessons (at least most of them). 

In other words, as pointed out by (Rovelli, 2020), this is a possible lesson from our endeavour to understand 
the interpretation of quantum mechanics. The best way to learn is to interact with the world while seeking to un- 
derstand it, readjusting our mental schemes to what we encounter and find, and trying to distinguish the signal 
and the noise (Silver, 2012). This conclusion is valid not only for science education. 
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Abstract. Scientific literacy is decreasing 
in the Czech Republic and inquiry-based 
science education (IBSE), believed to 
enhance it, is used rarely. Meanwhile, 
outdoor teaching is on the rise, but it is 
unclear if it could help promote inquiry. This 
study examines the frequencies of outdoor 
teaching and IBSE and their relationship 

in self-reported practice among motivated 
educators. Two questionnaires were 

filled out by educators who downloaded 
teaching resources in 2019 (N = 796) and 
those who enrolled in an online course 

in 2020 (N = 745). A high proportion of 
educators used some form of IBSE. Many 
of them unexpectedly often involved 
students in research design as part of 

IBSE. The frequency of student-designed 
investigation was correlated with the 
frequency of outdoor teaching in both 
samples. Both frequencies increased after 
the online course. This indicates that 
involving students in research design 

can be supported indirectly and that 

new professional development courses 
could use the attractiveness of outdoor 
education. Two groups of active teachers 
were identified, one focused on inquiry in 
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colleagues to teach outdoors. As they were 
represented more among primary school 
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Introduction 


There is an increasing group of educators, both schoolteachers and 
home-schooling parents, who devote their time and energy to learning 
outside the classroom. This trend is observable both in Central Europe and 
elsewhere worldwide (Cinéera & Holec, 2016; Waite et al., 2015). In various 
studies, diverse forms of learning outside the classrooms have been distin- 
guished, such as outdoor learning (including fieldwork, outdoor adventure 
activities, and schoolyard or community projects in Rickinson et al., 2014) and 
outdoor play and learning in natural environments (Gill, 2011). Some authors 
focus only on school-based outdoor learning, however, outdoor learning is 
part of education in non-formal settings (e.g. home-schooling, education in 
museums, botanical gardens or zoos) as described in Malone (2008). 


Outdoor Teaching on the Rise 


The rise of outdoor teaching and learning at schools in the Czech Re- 
public follows the examples from abroad (such as English Forest Schools or 
Danish “udeskole”). It lags behind the increase in the number of forest kin- 
dergartens, which has been inspired by German or Scandinavian models. In 
the Czech Republic, there were over 150 forest kindergartens in 2023 (https:// 
www.lesnims.cz/lesni-ms/historie-lesnich-ms.html), but only dozens of forest 
primary schools. While outdoor teaching is well established in alternative 
and private schools, the majority of schoolteachers in state schools in the 
Czech Republic use it rather rarely. Nevertheless, the proportion of schools 
that report using schoolgrounds for education increased from 60% in 2016 
to 65% in 2019 (Cinéera & Kroufek, 2021). 

There is an increasing number of teachers seeking inspiration for out- 
door learning on the online platform Ucimesevenku.cz (English version at 
https://lessonsingrass.com) and a related Facebook group with 10,3 thousand 
members (Nepraé & Sikulova, 2021). The accurate proportion of teachers 
and their frequency of using outdoor education is not known, but in the 
sample of 119 schools addressed by Vacha and Petr (2013), 46% of schools 
use the school garden for teaching regularly. Nevertheless, frontal education 
inside the classroom is reported as the dominant form in science classes at 
the lower-secondary level (Novosak et al., 2024), probably due to the same 
obstacles as described by Rickinson et al. (2004). The barriers include: (i) 
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concern about health and safety; (ii) lack of confidence in outdoor teaching; (iii) limiting opportunities for outdoor 
learning due to curriculum requirements; (iv) lack of time, resources, and support. 

Outdoor education is generally appreciated for different benefits such as health and wellbeing (Chawla, 2015), 
improved academic performance in different subjects (Becker et al., 2017), science process skills, students’ long- 
term memory and higher-order learning (Rickinson et al., 2014), confidence, self-esteem and social skills, leadership 
potential, and environmental responsibility (Chan et al., 2016; Malone, 2008). It is performed by primary school 
teachers quite often not only for teaching science in connection with nature, but also for various other subjects. 
This can be assigned to the category “learning in natural environments” rather than “outdoor education”. This is 
clear from the feedback to teaching materials from teachers on the Czech online platform for outdoor learning 
Ucimesevenku.cz, where the teachers appreciate the materials for Maths and Languages outdoors. 

The rationale for outdoor teaching remains vague for many teachers. As teachers usually do not verbalize their 
own purposes for choosing learning out of school, they often lack arguments dealing with benefits for students 
that could be used in debates with parents, colleagues, or headmasters. Teachers may also use general arguments 
not tightly aligned with their particular curricula. Czech teachers have an opportunity to find arguments regarding 
the benefits of learning out of the classroom in the publication Tajemstvi Skoly za Skolou [The Secret of the School 
behind the School] (Danis, 2018) or through online courses at Ucimesevenku.cz. These are grouped by researchers 
(Waite, 2020) as Healthy Bodies and Positive Lifestyles; Social, Confident and Connected People; Creative and Self- 
Regulated Learners; Effective Contributions and Collaboration; Care for Others and the Environment. The impact of 
teachers’ beliefs on their practice during a professional development program on outdoor learning was analyzed 
for secondary teachers in England (Glackin, 2016). Teachers who successfully taught outside generally held social 
constructivist beliefs about learning and valued opportunities that allowed their students to conduct ‘authentic’ 
science. Teachers who were less successful in teaching outside held traditional learning beliefs and appreciated 
the outdoors because it was a new and entertaining learning environment for their students. 


Inquiry Science Education and Scientific Literacy 


When focusing on opportunities to develop students’ science process skills, teachers may take advantage of 
inquiry-based learning, which has been very common in English-speaking countries for more than 30 years and 
has been widely promoted in educational practice, especially in science education (Furtak et al., 2012; Hmelo- 
Silver et al., 2007). In inquiry-based science education (later on IBSE), students are exposed to authentic problems 
or natural phenomena in real-world contexts — they are expected to acquire knowledge through observations, 
explanations, and experiments with real-world problems (Bybee, 2002; de Jong et al., 2013; National Research 
Council, 2012; Pedaste et al., 2015). 

Although promoted internationally through EU recommendation reports and dissemination projects, the 
impact of IBSE on scientific literacy (e.g. measured by PISA) is still debated (McConney et al., 2014; Sjaberg, 2018), 
however, it has been shown as effective in supporting students’engagementin science. The relationship between the 
frequency of IBSE and results in scientific literacy is not straightforward. When PISA’s composite variable (IBTEACH) 
was used, there even was a negative correlation between the level of inquiry and scientific literacy scores. However, 
when the IBTEACH index was disaggregated and each questionnaire item was mapped separately, the results 
showed contrasting and non-linear patterns (Oliver et al., 2019). The non-linear relationship can be illustrated by 
these items: Students who reported spending time in the laboratory doing experiments in some or most lessons 
performed better than students who experienced it in every lesson or never. The same is true for “drawing a con- 
clusion about an experiment they performed” and “doing an investigation to test ideas”. For the item “students are 
allowed to design their own experiments’, the pattern is non-linear, with high levels of scientific literacy linked 
to the low frequency of this item but with declining scores when this item is reported as “never or hardly ever”. 

The authors underscore the question of optimal frequency, quality, and purpose of different inquiry activities. 
The quality of IBSE is naturally what matters most, but this can only be assessed during classroom observation 
(Kotulakova et al., 2022; Oliver et al., 2019). Among various inquiry activities, designing and performing investiga- 
tions and drawing conclusions are the most important factors linked to scientific literacy. 

In the Czech context, using IBSE is observed as very rare; less than 4% observed lower secondary science 
lessons in the report of the Czech School Inspectorate (Ceska Skolni inspekce, 2018). Radvanova et al. (2017) have 
found a significant increase in the use of IBSE in the sample of upper secondary teachers from 3.43% of lessons 
in 2012 to 6.82% of lessons in 2017. Similarly, Cinéera and Kroufek (2021) have stated that 6% of teachers report 
using the whole inquiry cycle in their practice in a 2019 survey. 
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Czech school lessons are dominated by the transmission approach in science classes (Novosak et al., 2024), 
IBSE frequency has not been mentioned in this report at all. Recommendations focused on different aspects of 
“active learning’, which can be directly linked to IBSE as measured by PISA. Most students have rarely or never the 
opportunity to promote their science process skills such as working with graphs and tables, designing and con- 
ducting observations and experiments, or interpreting results. The report has revealed that most students have 
difficulty with designing the experiment. The report has not proved higher levels of scientific literacy among stu- 
dents who experience active learning (students’ experiment, own observation of natural phenomena and events, 
work with error), but the frequency of active learning positively influenced their attitude to science. Similarly, in 
the UK, Ofsted (Office for Standards in Education, Children’s Services and Skills) has encouraged the use of practical 
work in school science, criticizing “poor opportunities for students to plan, carry out, and evaluate investigations 
independently” (Ofsted, 2013, p. 10). 

According to Novosak et al. (2024), the teachers would appreciate practically oriented support for their peda- 
gogical work (e.g. teaching supplies and kits and methodological guides, worksheets, suggestions for experiments) 
and higher overall coherence of science education (e.g. continuity of the curriculum across the educational stages, 
cross-curricular and literacy links within topics). The report has concluded that professional development of teach- 
ers is essential for the introduction of innovations in science education. 


Professional Development for Outdoor Education and IBSE in the Czech Republic 


As professional development in the Czech Republic does not follow any rigorous plan, credit system, or 
certification, it is rather optional. Teachers use different opportunities, such as courses organized by NGOs and 
universities, usually only short-term ones. Online courses have been quite rare in this field until recently. The online 
course “U¢ime se venku” (Teaching Outdoors) has been one of the first of this kind in the Czech Republic (piloted in 
2019/20), later other online courses for science education (e.g. for IBSE or climate change education) have followed. 

In the Czech Republic, both outdoor teaching and using IBSE are promoted by NGOs (such as TEREZA, Educa- 
tional Centre) developing teaching materials made available online and promoted through social media. There is 
still a difference between the numbers of teachers and other educators interested in outdoor teaching (a Facebook 
group with ca. 10.3 thousand participants in March 2024) and IBSE (5.4 thousand); the number of teachers joining 
these Facebook groups has been increasing steadily. The IBSE approach is introduced to students at universities 
but with limited success, as they often do not have the opportunity to implement the method under the supervi- 
sion of experienced teacher mentors during their studies and later on as novice teachers. An innovative platform 
called Hyperspace (hyperspace.cz) is focused on implementing inquiry-based teaching and formative assessment at 
primary and lower-secondary level. It contains methodological materials, videos from real lessons, and pedagogical 
diagnostic tools (Stuchlikova et al., 2023). 

According to a report of the Czech School Inspectorate (Novosak et al., 2024), the majority of lower secondary 
science teachers consider collaboration with colleagues as most useful for their professional development, 20% 
of teachers value professional development courses, and only 10% esteem online courses. At the same time, the 
majority of teachers in the Czech Republic do not have any experience with reflective practices within school pro- 
fessional learning communities (Smetackova & Vozkova, 2021) and thus the real benefits from their collaboration 
with colleagues remain questionable. 

Introduction of both outdoor teaching and IBSE requires changes in educational strategies, classroom man- 
agement, or the role of teachers. The teachers search for opportunities to enhance their self-efficacy, gain sup- 
port, arguments, resources, and best practice examples to ascertain themselves and convince their colleagues, 
principals, and parents that such a change of educational methods is beneficial to the students. Cooperation with 
colleagues is considered a part of teachers’ self-efficacy (Bandura, 2007). Tschannen-Moran and Woolfolk Hoy (2001) 
have defined it as the teachers’ beliefs in their own abilities to effectively teach a subject to the students, ensure 
student participation and achieve the desired results. Researchers also add the component of collaborating with 
colleagues and parents and participating in the school management (Yang & Wang, 2019; Smetackova et al., 2017). 
In the previous study, primary school teachers were more active than lower secondary school teachers both in 
the frequency of outdoor teaching and using IBSE, in collaborating with colleagues, and in influencing decisions 
within the school management (Cihakova, 2021). 


601 


https://doi.org/10.33225/jbse/24.23.599 


Journal of Baltic Science Education, Vol. 23, No. 4, 2024 


INQUIRY AS A NATURAL PART OF OUTDOOR TEACHING: INSIGHT FROM MOTIVATED ISSN 1648-3898 | /erint, 
EDUCATORS USING ONLINE GOURSE AND RESOURCES IN THE GZEGH REPUBLIG 
ig ISSN 2538-7138 sontine/ 


Research Aim and Research Questions 


As outdoor teaching about nature brings up new students’ questions naturally, it is interesting whether the 
higher frequency of outdoor teaching promotes some aspects of inquiry-based education, specifically whether it 
increases the frequency of students’ questions and opportunities for students designing their own research. Us- 
ing self-reported behavior of teachers in questionnaires and the limitations of this approach is discussed below. 
Opportunities for students to formulate research questions and design their own research may be reported by 
teachers who use IBSE deliberately and also by teachers who have not attended any formal course on IBSE, or those 
who respond that they do not know the IBSE approach at all. 

However, it is not clear whether there is a group of teachers who use both approaches (outdoor teaching and 
IBSE) intentionally, as only those could serve as ambassadors of both outdoor teaching about nature and promot- 
ing the IBSE approach among their colleagues in school learning communities. Therefore, specific questions on 
teachers’ self-efficacy (subscale: collaboration with colleagues) and the number of colleagues in the same school 
practicing outdoor teaching were included. 

The impact of the online course from questionnaires (pre-test and delayed second post-test) for the online 
course“Ucime se venku” taking place in 2020 on the frequency of outdoor teaching and IBSE was evaluated. Another 
set of data from an online questionnaire filled out in 2019 by the teachers who downloaded teaching materials 
from the “Ucimesevenku.cz” website (designed by TEREZA, Educational Centre) was analyzed. 

In this study, the following questions were addressed: 

RQ1. Does the online course increase the frequency of outdoor teaching? 

RQ2. Does the online course increase the frequency of IBSE? 

RQ3. Is the frequency of using IBSE correlated with the frequency of outdoor teaching? 

RQ4. What are the characteristics of teachers regarding outdoor teaching, using IBSE and influencing 
other colleagues — are these traits correlated among primary and lower-secondary teachers? 


Research Methodology 
General Background 


The research was carried out as an evaluation of an online course that was piloted in the spring of 2020. It was 
offered as a voluntary free professional development program for school teachers and other educators (N = 745) 
at the Ucimesevenku.cz platform. The same website offered free teaching resources focused on outdoor educa- 
tion with inquiry elements. Educators (N = 796) were asked to fill out the questionnaire before downloading free 
resources (February — November 2019). Post-questionnaires for the online course were distributed in 2021. As it 
was still during Covid restrictions when schools in the Czech Republic were closed, the teachers self-reported only 
the activities they used during online education. In this study, only data from delayed post-questionnaires from 
the school year 2022/23 were analyzed. 

The study was conducted as action research with the aim of improving both the online course and teaching 
resources and tailoring other offered professional development activities (new online courses, webinars, Facebook 
group challenges, etc.) for the targeted audience. Another objective was to identify experienced teachers to address 
them as ambassadors of both outdoor teaching and IBSE and to reach them to participate as tutors in new courses. 

Online questionnaires of self-reported teachers’ behavior were used, and the answers were categorized for 
quantitative analysis. The correlation between the frequency of IBSE and outdoor teaching was analyzed. The 
impact of the online course was examined for different items separately using exploratory multivariate analysis to 
find the characteristics of different groups of teachers active in IBSE and outdoor teaching. 


Sample 


Two different datasets were used for this study. Both were samples of educators interested in outdoor teach- 
ing. Online Google questionnaires were used for addressing teachers on the website Ucimesevenku.cz. Answers 
were obtained from respondents downloading free teaching materials from Ucimesevenku.cz in 2019 (N= 796), 
later called the DOWNLOAD dataset. 

Further, responses from educators enrolling in the online course “Ucime se venku” in 2020 (N = 745) were 
used, later called the ONLINE COURSE dataset. The online course was focused on arguments about the benefits 
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of outdoor education, and it showed how to acquire or create and use simple equipment for outdoor education. 
To assess the impact of the online course, two post-questionnaires were distributed. However, because of school 
closures during the Covid pandemic, only the data from the delayed post-test that was distributed in 2022/23 
(N = 74) were used in the analysis. 

For the questions regarding the school practice, only subsets containing schoolteachers were used (RQ1, RQ2, 
and RQ4). Answers from the whole sample (including other educators) were used for the correlation between the 
frequency of outdoor teaching and using IBSE and the frequency of involving students in the investigation design. 

Specifically, there were 278 primary school teachers (ISCED 1), 144 lower-secondary school teachers (ISCED 2), 
and 436 other educators (such as home-schooling parents, preschool and early childhood educators, educators in 
non-formal settings) in the DOWNLOAD dataset. “Elementary” school teachers (the term is used synonymously with 
“primary” school teachers throughout the text) correspond to ISCED 1, and“lower secondary” correspond to ISCED 2. 

There were 252 primary school teachers, 107 lower-secondary school teachers, and 358 other educators in the 
ONLINE COURSE dataset. In the delayed post-questionnaire in 2022/23, only 98 participants sent their responses 
(74 teachers). Only this subsample of teachers was used for further analysis of the impact of the online course on 
the frequencies of different teaching strategies. The results therefore show the impact on the highly motivated 
teachers, as discussed below. 


Instrument and Procedures 


The questionnaires were designed to assess the effect of the course and to reveal the potential correlation 
between the frequency of outdoor teaching and using IBSE before and after the online course or using provided 
teaching materials. Items have been adopted from Smetackova et al. (2017) for self-efficacy and from the Czech 
School Inspectorate survey for frequency of outdoor teaching and IBSE steps to be comparable with the surveys 
at the state level. Both questionnaires included a subset of identical questions (on the frequency of outdoor teach- 
ing and involving students in investigation design) and other questions unique for each dataset. The DOWNLOAD 
dataset focused on teachers’ practice, frequency of different IBSE steps (see below), and other items not included 
in this analysis (such as teachers’ experience in outdoor teaching during their university study, use of different 
environments for outdoor teaching, or self-declared knowledge of IBSE principles). The ONLINE COURSE dataset 
included questions on teachers’ self-efficacy and planning. There were identical questions in the pre-questionnaire 
and the delayed post-questionnaire. The post-questionnaire contained additional questions on participation in 
other online courses of other professional development programs concerning IBSE in the due course. 

Responses were categorized as follows: frequency of outdoor teaching (in DOWNLOAD dataset): 0 = never, 1 = 
rarely, 2 = 2x per term, 3 = monthly or more often, 4 = weekly; frequency of outdoor teaching (in ONLINE COURSE 
dataset) : 0 = never, 1 = rarely, 2 = 2x per term or more often, 3 = monthly or more often; frequencies of IBSE steps, 
such as involving students in investigation design (later as Students’ design frequency, IBSE_design in Figure 5), 
frequency of students formulating questions (IBSE_questions), recording data and drawing a conclusion: 0 = never, 
1 = rarely, 2 = 2x per term or more often, 3 = monthly or more often. 

Other responses were categorized as follows: Teacher plan (“I prepare a year plan for outdoor teaching ac- 
cording to observable nature phenomena’: 4 = strongly agree, 3 = agree, 2 = disagree, 1 = strongly disagree), Argu- 
ments (“I know why to teach outdoors... but | don’t talk about the reasons because I’m not sure how to persuade 
others.’ = 0, “and | know where to find arguments to persuade others” = 1, “and | share the arguments with some 
colleagues” =2, “and | use them regularly when talking to other colleagues, parents or school management.’ = 3) 

The subscale “Influence on School Management” from the Czech teachers’ self-efficacy scale created by 
Smetackovd et al. (2017) was applied to assess the influence of teachers on teaching strategies used in their school 
(such as IBSE or outdoor education) and their potential to change the practices of their colleagues. This subscale 
was adapted from the original Bandura’s teacher efficacy scale (Bandura, 1997) consisting of questions with a 9 
point scale. Smetackova et al. (2017) used a 5-point frequency scale anchored with the expression of frequencies 
ranging from “never” to “always” that was piloted in a quantitative study (N = 583). The items included the following 
statements: | am able to express opinions on an important issue (i), |am able to contribute to solving a problem (ii), 
lam able to influence decisions at school (iii), |am able to enforce change in school practice among colleagues (iv). 
For this study, the variable labeled “Efficacy” was calculated as average from these 4 items, coded as follows: never 
= 0, rarely = 1, sometimes = 2, often = 3, always = 4. The answer “I am not able to decide” was coded as a missing 
value; teachers who did not respond to one or more questions were excluded from the analysis. 
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Data Analysis 


All variables used in the analysis came from self-reporting data in questionnaires and, therefore, are of ordinal 
character. Intervals between individual values were assumed to be roughly comparable, and their values were 
therefore analyzed as numeric (quantitative) variables with normal errors. Changes between their values before 
enrolling on the online course and after it were compared by an paired two-sided t-test. The correlation between 
outdoor teaching and the frequency of involving students in investigation design and teaching at a primary school 
was examined using a simple (regression) linear model to express shared variation between these variables using 
R? and to perform a significance test using F-statistic. 

A number of variables for individual teachers were collected, and therefore mutual interrelationships across 
the whole set of measured variables could be examined. Such mutual correlations among the teachers’ responses 
were identified using Principal Component Analysis (PCA) of their correlation matrix. The PCA results showed the 
amount of shared variation (proportion of variation accounted for by the first and second principal axes of varia- 
tion) and the contribution of individual variables to these principal axes (variable scores). PCA was calculated using 
package vegan (Oksanen et al., 2020, ver. 2.5-7) from the R environment (R Core Team 2021). 

PCA was also used to summarize teachers’ responses in the four variables, capturing the frequency of IBSE steps 
(i.e. formulating questions, involving students in investigation design, recording data, and drawing a conclusion). 
As all these variables are expressed on the same scale, PCA on covariance matrix, i.e. without standardization was 
used. Here, the first axis accounted for 75.3% of the total variation in these four variables, indicating high correla- 
tions among them. This permitted the use of the teachers’ score of the first axis of this PCA as a representative value 
summarizing most of the relevant variation in these four variables (further called IBSE score). The correlation of this 
IBSE score and outdoor teaching frequency was again analyzed using a linear model. All analyses were conducted 
in the R environment, version 4.1.2 (R Core Team, 2021). 


Research Results 


After the online course, both the frequency of outdoor teaching and the frequency of student-designed 
research reported by teachers increased. Teachers’ planning of outdoor teaching activities and their knowledge 
of arguments for outdoor teaching increased as well. However, reported teachers’ self-efficacy with respect to 
influencing colleagues and the school as a whole decreased. Nevertheless, 8 teachers reported higher teachers’ 
self-efficacy after the course and 13 teachers reported an increased number of colleagues in their school outdoor 
teaching (ONLINE COURSE dataset). 

The frequency of IBSE: 29.6% of teachers gave students the opportunity to design their own investigation 
twice per term, and 11.6% of teachers did so monthly. The results were similar for primary and lower-secondary 
school teachers. 15.8 % of teachers taught outdoors monthly or more often (ONLINE COURSE dataset). 

In the dataset of participants downloading teaching materials, 26.5 % of teachers taught outdoors monthly, 
29% of teachers gave students the opportunity to design investigation twice per term, and the other 25.2% of 
teachers did so monthly. Among other educators, the frequencies were higher: 44% of educators taught outdoors 
monthly or more often (DOWNLOAD dataset). 

Completing the online course increased the frequency of outdoor teaching among teachers in delayed post- 
test (Figure 1). The frequency of outdoor teaching is coded: 0 = never, 1 = rarely, 2 = 2x per term or more often, 
3 = monthly or more often; t (41) = -2.231, p = .031. No teacher declared that he/she would never teach outdoors. 
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Figure 1 
Outdoor Teaching Frequency Before the Online Course and 3 Years Later 
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Note. The values of the y axis have been jittered to improve visibility. 


Completing the online course increased the frequency of student-designed research reported by teachers in 
delayed post-test (Figure 2): t (43) =-2.466, p = .018. The frequencies are coded as 0 = never, 1 = rarely, 2 = 2x per 
term or more often, 3 = monthly or more often. Completing the online course also increased the knowledge of 
arguments for outdoor teaching (ONLINE COURSE dataset). 


Figure 2 
Student-Designed Research Frequency Before the Online Course and 3 Years Later 
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Correlation between inquiry frequency (measured as IBSE score, i.e. the score at the first axis of PCA summariz- 
ing variation in four variables measuring frequencies of different IBSE steps) and outdoor teaching frequency was 
found in the DOWNLOAD dataset (Figure 3). The correlation was significant using ANOVA (F = 173.75; df= 1,794; p< 
.001) and explains almost 18% of the variation in the IBSE score (Adjusted R2 = 0.1785). Outdoor teaching frequency 
is coded as: 0 = never, 1 = rarely, 2 = 2x per term, 3 = monthly or more often, 4 = weekly. 

Correlation between inquiry frequency (measured as IBSE score) and outdoor teaching frequency was found 
in the ONLINE COURSE dataset as well for all educators (N = 718) and school teachers only (N = 360). The correla- 
tion was significant using Spearman's rank correlation coefficient, for all educators (p = .496, p < .001) and school 
teachers only (p = .389, p < .001). These results are not shown. 


Figure 3 
Relationship between IBSE score and Outdoor teaching frequency (N = 796) 


0.4 7 


0.2 4 


0.0 4 


IBSE score 


0.2 4 


0.4 4 


T T T T T 
0 1 2 3 4 


Outdoor teaching frequency 


Outdoor teaching frequency in the DOWNLOAD dataset was correlated with a particular IBSE step: student- 
designed research frequency (Figure 4). The correlation was significant using Spearman's rank correlation coefficient 
(ep = .454, p <.001). 

The optimal frequency of outdoor teaching that yielded the highest frequency in student-designed research 
was monthly or more often, dropping with a more regular frequency (weekly). 


Figure 4 
Relationship between Student-Designed Research Frequency (step of IBSE) and Outdoor Teaching Frequency (N = 796). 
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Teachers’ characteristics from the delayed post-questionnaire after the online course regarding outdoor 
teaching, using IBSE, involving students in formulating research questions, planning, teachers’ self-efficacy and 
influencing other colleagues were analyzed. Principal Component Analysis (PCA) showed that elementary school 
teachers were more active in all of the following characteristics. The first and the second axes explained 49.2% 
variation and 16.8% variation, respectively. The first axis divided teachers according to their activity or frequency 
in all variables. The second axis divided teachers into two groups, one focused on IBSE steps in their own practice 
and the other on using arguments to convince colleagues to teach outdoors more often (Figure 5). 


Figure 5 
Principal Component Analysis of Teacher Responses in Delayed Post-Questionnaire (N=74). 
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The frequency of both outdoor teaching and using IBSE in the sample of teachers downloading materials is 
comparable to other surveys from the Czech Republic, although the sample came from teachers who were highly 
motivated. 26.5% of teachers who downloaded the teaching materials teach outdoors monthly. This is similar to 
Neprag and Sikulova (2021), who found out that more than 30% of the sample of 120 primary school teachers 
teach outdoors regularly, and 80% reported that they would like to do it more often. Only 14% of primary school 
teachers entering the online course reported that they teach outdoors monthly. They were probably motivated to 
teach outdoors more often and that is why they enrolled in this voluntary professional development. 
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The Czech School Inspectorate (Ceska Skolni inspekce, 2018) has found a very low frequency of inquiry- 
based lessons, less than 4% of observed lessons. Although IBSE was not the main focus of the online course 
and teaching materials on the website Ucimesevenku.cz, the community of teachers interested in both outdoor 
teaching and inquiry overlaps, so we could presume the respondents to be more active in both. A correlation 
between the frequency of outdoor teaching and using IBSE or its aspect (student-designed research) was found. 
In the study, 25.2% of teachers who downloaded the teaching materials offered the opportunity for student- 
designed investigation on a monthly basis, while only 11.6% did so among teachers enrolling in the online course. 
The Czech School Inspectorate (Ceska skolni inspekce, 2018) researched science lessons, and according to the 
self-reported frequencies of teaching practices, only 16% of teachers claimed that their students performed an 
experiment that they would design themselves monthly. 

Cinéera and Kroufek (2021) found that only 19% of teachers see students as partners in the inquiry process 
and allow them to participate actively in the decisions about the design of the research, and only 5% of teachers 
attribute a leading role during inquiry to the students. In the survey on environmental literacy (Novosak et al., 
2020), 32% reported using IBSE; from this subset, only 30% of teachers involved students in deciding about the 
inquiry process as a partner. It was found that 36% of teachers report using IBSE when enrolling in the online 
course, 11.6% report enabling students to design investigations monthly, and 29.6% do so twice per term. The 
teachers (DOWNLOAD dataset) in the study allowed students to design investigation monthly (25.2%) and twice 
per term (29%) which is clearly more frequent than in the Czech School Inspectorate surveys mentioned above. 
Nevertheless, it cannot be assessed if teachers see the students as leaders or only just participants in the deci- 
sions because the teachers’ beliefs were not addressed in the questionnaires. 

In the study of the self-reported frequency of IBSE in relationship to teachers’ beliefs and their science 
teaching self-efficacy (Lucero et al., 2013), authors revealed a very low frequency of student-designed research 
among teachers in rural Ecuador. The teachers preferred structured inquiry. It was difficult for teachers to give 
up control, teachers did not ask students to formulate their own research question and define, by themselves, 
the research procedure. This was linked to their context beliefs: Teachers who believed that effective science 
teaching was possible in their schools (thanks to supportive colleagues and school management, appropriate 
scientific equipment etc.) also believed that effective performance was possible for students. The teachers from 
the present study actively sought support through the online community at Ucimesevenku.cz, and many of 
them supported their colleagues. 


Impact of the Online Course and Potential Ambassadors of Outdoor Teaching and IBSE 


The approach of the online course participants to student-designed research is more positive than among 
the teachers from the Czech School Inspectorate surveys. In the group of all educators (school teachers, home- 
schooling parents, preschool and early childhood educators, educators in non-formal settings), 28% of educators 
have declared that they offered this opportunity monthly before they enrolled in the online course (whereas only 
11% teachers did so) and 33% educators do so after the online course. This rather high proportion among active 
online course participants is consistent with the findings about teachers’ beliefs by Kotulakova (2021). It revealed 
that Slovak teachers enrolling in the professional development course in IBSE highlight that they perceive their 
students as curious and active independent researchers and thinkers, see themselves as providers of a stimulat- 
ing environment for cooperation, and acknowledge that students must construct their knowledge individually. 

A correlation between the frequency of outdoor teaching and using IBSE or one of its aspects (student- 
designed research) was found. Two groups of active teachers after the course were identified, one focused on 
IBSE activities in their own class (outdoor inquiry enthusiasts) and the other influencing other colleagues and 
motivating them to teach outdoors more regularly using arguments retrieved from the online course (teaching 
outdoors ambassadors). Both groups were represented more by primary school teachers; participating lower- 
secondary school teachers were not so active regarding the frequency of outdoor teaching, convincing others, 
or using IBSE in their own class. This is in line with previous analysis of entry questionnaires prior to the course 
published in Cihdkova (2021), with primary school teachers playing a more active role in influencing their col- 
leagues. 

The active role of primary school teachers in IBSE is rather unexpected as primary school teachers tend to 
worry about the extent of their personal knowledge of the science content, have lower self-confidence in teach- 
ing science in general and using IBSE in particular (Harlen, 2021). However, at the same time, their pedagogical 
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knowledge enables them to encourage their students to attain inquiry skills. Primary school teachers have 
fewer obstacles (organizing lessons within a schedule, etc.) than lower-secondary teachers, and they have the 
pedagogical training and experience needed to organize group work that is useful for learning in outdoor set- 
tings. Primary school teachers usually experience difficulties in guiding students through the process of inquiry 
(Zion et al., 2007), regarding the formulation of a research question and the design of an investigation (Yoon et 
al., 2012). Nevertheless, the sample reported in the present study of highly motivated primary school teachers 
received support in the online course, used available teaching resources from the website Ucimesevenku.cz, 
and was inspired by other members of the Facebook group. According to their self-reported behavior, they were 
able to use the advantages of outdoor teaching for inquiry. 

The opportunities for student-designed investigations are emphasized in the present study in accordance 
with the definition of inquiry (NRC, 1996) as involving: “making observations; posing questions; planning in- 
vestigations; reviewing what is already known; using tools to gather, analyze, and interpret data; proposing 
answers; explanations and predictions and communicating the results.’ Therefore, teachers’ role consisting of 
“asking students to make predictions and plan investigations” (Harlen, 2021) was addressed, although this step 
was reported rather rarely in other studies and surveys, and it is limited to lessons with the guided or open in- 
quiry. Results from the questionnaires are based on self-reported behavior of teachers and may not reflect the 
reality correctly. However, they indicate that student-designed research is a part of the practice of the sample 
of highly motivated teachers. 

This is in contrast to a recent study of video-taped inquiry-based lessons of primary and lower-secondary 
teachers from 20 Norwegian schools (Kersting et al., 2023) that coded the behavior of teachers through differ- 
ent phases of the inquiry cycle. It revealed that in none of the 73 observed lessons students planned investiga- 
tions based on their own questions or predictions and that both primary and lower-secondary school students 
barely developed their own questions and hypotheses. Although the sample of teachers was selected from the 
schools regularly collaborating with university researchers which would implicate higher than average quality 
of inquiry, a high proportion of observed lessons did not include any elements of inquiry. In the inquiry lessons, 
the primary data was often collected according to a given recipe when students did practical work. All the vid- 
eotaped lessons took place inside the classroom, and this could influence the results, as it is extremely difficult 
to videotape lessons outdoors, but this limitation is not mentioned by the authors. 

Nevertheless, the authors (Kersting et al., 2023) compared the quality of IBSE between primary and lower- 
secondary schools summarizing that primary school students collected and documented data more systematically 
than lower-secondary students and that consolidations of results were slightly more emphasized and of higher 
quality at the lower-secondary than at the primary level. This indicates the potential for mutual collaboration 
between primary and lower-secondary teachers that would be beneficial to both groups. 

Establishing a community of practice is an essential part of any successful teacher professional develop- 
ment program (Timperley et al., 2007). It cannot be fully substituted by online Facebook groups (as used in the 
online courses of TEREZA, Educational Centre), but it should be school-based or organized at a district level. 
Such professional learning communities are already widespread in the Czech Republic for language teachers 
(https://www.kellnerfoundation.cz/en/helping-schools-succeed) or for physics and chemistry teachers (Elixir 
do kol, https://www.elixirdoskol.cz). They are still missing for biology teachers in the Czech Republic, whereas 
they are widespread, e.g. in the USA, where Kelley et al. (2020) found that teachers who participated ina STEM 
community of practice increased their self-efficacy. 

Drawing from the results, we would recommend involving both primary school teachers and lower-secondary 
biology teachers to become members of such learning communities. It would be beneficial as the primary and 
lower-secondary stage is mostly part of the same institution in the Czech Republic, but teachers rarely meet 
between the two stages. Inviting more experienced teachers from other schools or districts as ambassadors of 
outdoor education and IBSE would be useful. Cooperation with external experts is an important part of effec- 
tive professional development, according to Timperley et al. (2007), and inspiration from colleagues from other 
schools can probably be accepted by their peers better than from external experts from universities or NGOs. 
That is why highly motivated teachers such as those enrolled in professional development programs (includ- 
ing online courses focused on IBSE phases or student assessment in outdoor education and IBSE) are needed. 

Moreover, it is important that lower-secondary teachers, with their specific needs, interests, and limitations, 
are inspired by their more experienced peers. The organizer of the Ucimesevenku.cz online platform (TEREZA, 
Educational Centre) has recently promoted online courses on IBSE lectured by lower-secondary teachers. They 
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address the same community of teachers (followers of the website and social media of Ucimesevenku.cz) and 
several participants of this study were enrolled or already participated in the IBSE online course when they filled 
the delayed post-questionnaire. TEREZA, Educational Centre provides new teaching materials, e.g. inquiry about 
climate change and carbon cycle. This is in line with the recommendation of Novosak et al. (2024) that Czech 
teachers would appreciate practically oriented support for their pedagogical work (e.g. methodological guides, 
worksheets, suggestions for experiments) to involve inquiry and enhance scientific literacy. Further research for 
evaluating the benefits of particular teaching resources is needed. 

It is striking that using outdoor education as a means for promoting inquiry is hardly ever mentioned among 
the recommendations for introducing IBSE through teacher professional development programs (further called 
PDP): Harlen and Allende (2009) recommended enabling teachers to use the environment and taking advantage 
of the range of contexts in which teachers learn inside and outside the classroom, while a more recent publication 
(Harlen, 2021) emphasized only the role of ICT, thought-provoking questions and scaffolding for initial teacher 
training and professional development in IBSE. This can be attributed to the fact that outdoor teaching per se 
can be challenging for an inexperienced teacher (Rickinson et al., 2004), and this could discourage participants 
from both outdoor teaching and IBSE if the PDP is not thoughtfully organized. 

When entering the combination of key words “outdoor AND (IBSE OR (inquiry AND education)) AND profes- 
sional development” into the Web of Science Core Collection, only 13 publications were found. Apart from the 
already cited research from the present author (Cihakova, 2021), these articles were related to ICT or mobile 
applications developed by teachers for students or were focused solely on inquiry in geography. 

Although outdoor education was recognized to support the implementation of the IBSE approach on a 
large scale (LOCM, 2006; Rocard et al., 2007), it did not result in wider adoption of PDPs. Outdoor Education or 
Learning in Outdoor Contexts was used in dissemination projects at the European, national or regional level, 
e.g. INQUIRY Project that took place in botanical gardens and nature history museums in 11 European coun- 
tries (Kapelari, 2015) and project Oborovy mentoring (Cihdkova, 2021) in the Czech Republic. However, as the 
results from these PDPs were not published in peer reviewed journals but rather as final project reports only, 
the potential of outdoor education for promoting IBSE is not recognized and widely exploited by researchers 
or teacher educators. 

The connection between outdoor education and IBSE is difficult to trace in the literature, probably due to 
diverse labels used for different forms of learning outside the classrooms, e.g. outdoor learning (Rickinson et al., 
2014), outdoor play and learning in natural environments (Gill, 2011) or learning in outdoor contexts (Kapelari, 
2015). In addition, it seems that the authors dealing with either science education or outdoor education (em- 
phasizing adventure or environmental aspects) publish in different journals and attend different conferences 
or international projects. 

Including outdoor teaching, at least for the first phase of the inquiry process, would be beneficial because 
inquiry starts with a question or problem raised by a new event or experience (Harlen, 2021) that happens 
outdoors naturally, both in school practice with students and during PDP for teachers. 


Limitations of Self-Reported Strategies and Behavior 


The results of the present study can only be generalized for outdoor teaching enthusiasts, as the sample 
used is retrieved from the questionnaires filled out by teachers visiting a specialized website and online course. 
Among teachers in general, the frequencies of outdoor teaching and IBSE are probably lower (Ceska Skolni 
inspekce, 2018; Novosak et al., 2024). 

As the principles of the IBSE approach were not clearly described in the questionnaire because it was part 
of one question (to assess how well it is understood by the respondents), it could result in overestimating the 
frequencies of particular inquiry activities. In the previous study, it was found that teachers report involving 
students in investigation design even though they do not have the knowledge of IBSE (Cihakova, 2021). This 
result is surprising, as this aspect of IBSE is usually not so easily adopted (Bernard & Dudek-Rdézycki, 2020; Lucero 
et al., 2013) and even after an extensive professional development program, teachers do not report it. 

The self-reported teaching practices in the questionnaires are generally overestimated (Kotulakova et al., 
2022; Monet & Etkina, 2008). Therefore, only questions regarding the frequencies of particular practices were 
included, and teachers were not expected to respond to more complex questions describing their skills or qual- 
ity of teaching practice. 
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According to Kotulakova et al. (2022), the areas that are most misinterpreted and overestimated by teachers 
include formulating research questions, analyzing data, and drawing conclusions, which the authors describe as 
the most effective processes in student learning. Similarly, when observing lessons in rural schools in Ecuador, 
authors compared the answers from the questionnaires with reality (Lucero et al., 2013) and found a complete 
lack of inquiry teaching. Specifically, there was no space for students’ questions (3 students’ questions vs. 1600 
teachers’ questions in 20 science lessons), while according to the questionnaires, structured inquiry was reported 
by teachers. 

This limitation was addressed in this study by asking the teachers to invite the researchers to their outdoor 
teaching lesson for observation and to participate in the interviews to compare the answers with the observed 
teaching practices (mainly IBSE - students’ questions and student-designed research). The phase of designing 
the investigations could be supported by teachers’ interventions as described by van Uums et al. (2016) in 3 
domains: Procedural - Scaffolding the procedure of formulating a research question with criteria and examples; 
Epistemic — Explaining design criteria of a proper investigation and questioning students about considerations 
regarding their research design, such as the number of research subjects and measurements; Social - Facilitating 
students’ collaboration, for example, by dividing roles such as chairman, and discussing individual responsibili- 
ties regarding these roles. 


Conclusions and Implications 


The present study showed that developing opportunities for active participation of students in the inquiry 
can be supported indirectly, through voluntary professional development (online course) focused on outdoor 
teaching. As outdoor teaching naturally offers the opportunity for formulating students’ own research questions 
and participation in the design of investigations, the future professional development courses should use this 
advantage and could address highly motivated teachers experienced in outdoor education. More research on 
combining IBSE and outdoor teaching would bring helpful insights for designing professional development 
programs for in-service teachers and pre-service teacher training. 

Active teachers, particularly primary school teachers, have proven to serve as ambassadors of both outdoor 
teaching and IBSE. It is important that lower-secondary teachers, with their specific needs, interests, and limita- 
tions, can be inspired by their more experienced peers. Hopefully, the current ongoing education reform of the 
state curriculum will highlight the need for change from a teacher-centered transmission approach to more 
student-centered forms of science education. The groups of teachers, such as outdoor inquiry enthusiasts and 
outdoor teaching ambassadors, may become the centers of future learning communities, where both primary 
and lower-secondary teachers could share their experience. More active lower-secondary teachers are needed 
to be educated in IBSE to serve as role-models for their peers. Professional development should also aim at 
enhancing their motivation and skills to share experience with their peers. 

When voluntary online courses in different aspects of IBSE (e.g. assessment or using ICT) are offered, we 
recommend combining them with outdoor teaching and aiming at experienced outdoor inquiry enthusiasts 
to enhance their teaching skills. Offering free teaching materials for beginners to download can probably also 
support the frequency and quality of IBSE, but more research on this topic is needed. 

However, unless the professional development of science teachers starts to follow mandatory rules and 
clear goals, the desirable changes in science lessons in Czech schools in general cannot be expected. 
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Abstract. Pedagogical Content Knowledge 
(PCK) is crucial for effective teaching, but 

it isa complex and implicit knowledge, 
especially in chemistry education. Therefore, 
this study explores the use of the LBCD 
(Learning by Collaborative Design) model 
to support PCK development among 
pre-service chemistry teachers. This study 
adopted a one-group pretest-posttest 
experimental design, conducting the 
intervention with 210 participants. After 
data collection, the study first conducted 
exploratory and confirmatory factor 
analyses on the pre-and post-test data, 
confirming the good reliability and 

validity of the designed chemistry PCK 
questionnaire. Additionally, paired sample 
t-tests were used to measure pre-service 
teachers’ PCK development in dimensions of 
CTO, KOA, KOC, KOL, and KOS. Furthermore, 
unstructured interviews helped to further 
clarify the participants’ views of the 
effectiveness of the LBCD curriculum. The 
results showed that the developed chemistry 
PCK questionnaire and the LBCD model had 
a significant statistical impact on the PCK 
elements development among pre-service 
teachers, although the development of 
these elements was not balanced. Finally, 
the study provides suggestions for the 
measurement of PCK elements, as well 

as chemistry teacher education, such as 
applying the LBCD model to different 
teacher groups and testing PCK element 
models using the methods of this study. 
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Introduction 


Pedagogical content knowledge (PCK) refers to the professional knowl- 
edge and skills that a teacher possesses in a particular subject or content area 
and is key to ensuring effective teaching. As Park and Oliver (2008) point out, 
pedagogical content knowledge is a core element of a teacher's professional 
competence. It includes an understanding of concepts, principles, theories, 
skills, and methods related to a specific discipline, as well as knowledge 
and skills related to the teaching process (e.g., understanding instructional 
strategies, teaching methods, student learning characteristics, assessment 
techniques, and classroom management), the interplay between content 
and instructional methods, and an awareness of student learning difficulties 
and misconceptions (Shulman, 1986; Shulman, 1987; Neumann et al., 2019). 
Teachers need to have a deep understanding of not only subject matter 
knowledge but also relevant pedagogical methods and skills. This integra- 
tion of ‘pedagogical content knowledge’ and ‘pedagogical methodological 
knowledge’ enables teachers to truly excel in their profession and provide 
quality educational services to their students (Castellano & Mikeska, 2023). 

Scholars have different definitions of PCK categorization (Berry et al., 
2015). Shulman (1986) initially identified two components of teacher exper- 
tise: content knowledge and (general) pedagogical knowledge. Magnusson 
et al. (1999) proposed the widely used five-component model of PCK, which 
includes orientation to science teaching (OST), knowledge of the curriculum 
(KoC), knowledge of the learner (KoL), knowledge of strategies (KoS), and 
knowledge of assessment (KOA) (Koberstein-Schwarz et al., 2022; Park, 2021). 
Gess- Newsome et al. (2019) hypothesized three internal structures of content 
knowledge (CK), pedagogical knowledge (PK), and contextual knowledge 
(CxK). Despite differences in the definition of PCK, scholars have emphasized 
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the importance of teacher expertise (Pilous et al.) It represents a deep understanding and mastery of the content 
taught in the classroom, bridges the gap between subject-matter expertise and instructional practice, and is an 
essential component of effective teaching and learning (Forsler et al.) By enhancing PCK, teachers can provide 
students with a more effective and impactful learning experience for students, giving them the knowledge and 
skills they need to succeed (Kleickmann et al., 2015; He et al., 2021). 


Research Problem 


PCK is a complex and implicit form of knowledge in teachers’ cognitive structures (Pilous et al., 2023). While 
teachers can perceive, experience, and develop PCK, expressing and formally transmitting PCK through language, 
text, or symbols poses challenges (Barendsen & Henze, 2019). Due to the implicit nature of PCK, the development 
and assessment of PCK have become challenging. Nevertheless, researchers have developed various PCK assess- 
ment tools, such as CoRe (Content Representations) and PaP-eRs (Pedagogical and Professional Experience Reports) 
developed by Loughran et al. (2007), PCK ERT (PCK Evidence Reporting Table) and PCK assessment standards 
developed by Park and Oliver (2008), and assessment tools developed by Leader-Janssen et al. (2013), TSPCK (PCK 
Evidence Reporting Table), and PCK assessment standards. Jang et al. (2013) explored pre-service teachers’ PCK 
through workshops, mid-term and final evaluations, and teacher interviews. Aydeniz and Kirbulut (2014) devel- 
oped an assessment tool for pedagogical content knowledge for secondary school science teachers. Kirschner 
et al. (2016) developed a paper-and-pencil test tool for physics teachers’ PCK. Gro&schedI et al. (2019) developed 
a tool to measure pre-service biology teachers’ pedagogical content knowledge using the Rasch model. These 
assessment tools mainly collect qualitative data through open-ended questionnaires, classroom observations, 
interviews, and two-stage questionnaires, with only a few researchers attempting to develop quantitative tools 
for evaluating teachers’ PCK. In chemistry education, there is no widely influential quantitative assessment tool 
for teachers’ PCK development. 

In addition, the development of PCK is a continuous process that requires ongoing professional learning and 
reflective practice (Nilsson & Vikstr6m, 2015; Tal et al., 2021). In chemistry education, pre-service teachers need to 
develop a solid foundation of chemistry content knowledge, pedagogical knowledge, and skills related to chem- 
istry teaching (Padilla & Van Driel, 2011; Ekiz-Kiran et al., 2021). They must understand effective teaching strategies, 
assessment methods, and classroom management techniques to promote student engagement, critical thinking, 
and active learning (Can-Kucuk et al., 2022). Researchers have proposed various intervention models to facilitate 
the effective development of PCK in authentic teaching contexts for pre-service chemistry teachers. Combined 
with the comprehensive PCK consensus model proposed by Hume et al. (2019), the paths followed by researchers 
in designing PCK intervention models can be roughly divided into three categories, as shown in Figure 1. The first 
category directly impacts the practice of PCK within personal PCK, such as case studies and students serving as 
cooperative teachers (Schultze & Nilsson, 2018). The second category directly influences the planned PCK within 
personal PCK, aiming to promote the development of chemistry teachers’ planned PCK, including instructional 
design, lesson planning, and curriculum development (Gess-Newsome et al., 2019). This can be achieved through 
curriculum analysis, lesson plan development, and peer collaboration (Nilsson & Vikstré6m, 2015). The third category 
focuses on the collective professional knowledge component of PCK, aiming to enhance the development of col- 
lective professional knowledge within the chemistry teacher community, thereby promoting the development of 
individual professional knowledge (including planned professional knowledge and enacted professional knowl- 
edge). Examples in this area include professional learning communities (PLCs) and collaborative CoRe (Content 
Representations) design with mentors (Hume & Berry 2013). Currently, the first two intervention paths are more 
frequent, while research on the third path is relatively limited. 
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Figure 1 
Pathways for PCK Development 
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Research Focus 


Based on these analyses, there is little research related to the third pathway of PCK development and there 
is aneed for a more standardized quantitative tool to assess the impact of intervention programs on pre-service 
chemistry teachers PCK. To address this problem, the intervention model of this study aimed to focus on the pathway 
of directly impacting collective PCK development to promote individual PCK development (Path 3). The Learning by 
Collaborative Design (LBCD) model is an innovative approach to education that emphasizes collaborative learning 
(Koehler et al.) The LBCD model emphasizes the creation of solutions to real-world problems through collaborative 
design, engaging learners in the active construction of knowledge (Stammes et al., 2020). In this model, pre-service 
teachers engage in the process of designing and solving problems in small groups, interacting, discussing, and 
collaborating. The LBCD model also encourages pre-service teachers to incorporate digital tools, simulations, and 
other resources into their teaching practices (Herodotou et al., 2022) to facilitate their collaborative learning and 
deepen their understanding of chemistry concepts (Yeh et al., 2021). Furthermore, to meet the need for a more 
valid questionnaire for quantitative measurement of PCK, this study developed a chemistry PCK scale based on 
the 5-element PCK model (Magnusson et al., 1999) and examined its quality. On this basis, this study measured the 
impact of LBCD on pre-service chemistry teachers’ PCK development and verified the effectiveness of curriculum 
design in promoting chemistry PCK development. 


Research Questions 


This study aimed to address the following research questions: 
1) Does the newly developed Chemistry PCK questionnaire, based on the 5-components PCK model, valid 
measure the PCK in pre-service chemistry teachers? 
2) Towhat extent does the LBCD curriculum effectively promote the development of PCK among chemistry 
pre-service teachers? 


Research Methodology 
Context 


This study employed a one-group pre-test and post-test research design to survey and intervene with pre- 
service chemistry teachers from a renowned teacher training institution in Guangdong, China, during the first 
semester of 2022. The institution focuses on preparing pre-service teachers for primary and secondary schools in 
China. The study was approved by the Chemistry Education Committee at the University. Before implementing 
the intervention, an advertisement was sent to all pre-service chemistry teachers. Participation was voluntary. 
Confirmed participants signed up with the lecturer and completed a consent form. During the intervention, the 
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preservice teachers participated in the LBCD curriculum. Before and after the intervention, participants were re- 
quired to complete a pre-and post-test PCK questionnaire. 


Participants 


Regarding the selection of the sample size for this study, since the study mainly involves an instructional in- 
tervention and adopts a pre-post questionnaire format, as well as conducting CFA and EFA analysis, the choice of 
sample size needs to consider multiple factors. Comrey and Lee (2013) proposed that for factor analysis, a sample 
size of 50 is considered very poor, 100 is poor, and 200 is acceptable, and this criterion has been widely used. Fur- 
thermore, Anderson and Gerbing (1984) conducted a Monte Carlo simulation and reached similar conclusions, 
indicating that when each latent variable has 3 indicators and the sample size is > 200, the probability of conver- 
gence failure is almost zero, and there are no incorrect solutions. Considering the current number of pre-service 
teachers of chemistry education in the university and the practical feasibility of the intervention, invitations were 
sent to all relevant students who had completed foundational courses in higher education chemistry, including 
inorganic, organic, analytical, physical, and structural chemistry, as well as education-related courses such as edu- 
cational psychology, but had yet to receive specific training in chemistry teaching methods, pedagogical skills, or 
advanced topics in chemistry education. Ultimately 210 pre-service chemistry teachers (aged 20-22 years) volun- 
teered to serve as participants, forming the formal sample size for this study. These included 108 females and 102 
males. Also, to further gain a specific understanding of the impact of the LBCD curriculum on PCK, 30 students 
were randomly selected to participate in unstructured interviews as supplementary data of students’ views of the 
effectiveness of the LBCD curriculum on their PCK development. 


Design of the LBCD Curriculum 


The Learning by Collaborative Design (LBCD) curriculum model comprises five phases, as depicted in Figure 
2: (1) Collaborative Design Analysis: Teachers provide resources related to the theory of chemistry teaching design. 
Working in groups, preservice teachers discuss PCK components and their connections in the context of specific 
chemistry teaching topics. (2) Collaborative Design Guidance: Facilitated by instructors, preservice teacher groups 
engage in classroom discussions and interactions to address doubts and questions about teaching design. (3) 
Collaborative Design Construction: Preservice teacher groups collaborate and divide responsibilities to search for 
and organize teaching resources related to a specific topic. They then discuss and construct the group's teaching 
design. (4) Collaborative Design Refinement: Preservice teacher groups sequentially present their group teaching 
designs in the classroom and make further revisions based on feedback from instructors and other groups. (5) 
Collaborative Design Reflection: Preservice teachers share their refined group teaching designs and present their 
learning reflections. These five phases are interconnected and form a continuous cycle of collaborative design- 
reflection-revised collaborative design-revised reflection (Brennan & Gorman, 2023). This iterative process aims to 
develop the collective PCK of preservice teacher groups and, in turn, promote the development of individual PCK 
among preservice teachers (Lantz-Andersson et al., 2018). 
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Development of Chemistry PCK Questionnaire 


The process from questionnaire development to formal measurement involved four steps. Step 1 is defining 
the Components and Content of PCK. In this step, the PCK five-component model was used as the theoretical 
framework (Aydin et al., 2014). 

The original Orientations to Science Teaching (OST) mainly refers to the field of science education, in this study, 
we are focusing on chemistry pre-service teachers, so it has been modified to Chemistry Teaching Orientation 
(CTO) to be more in line with the subject matter. The revised specific content of the five components is as follows: 


Table 1 
Contents and Dimensions of the PCK 


Dimensions Descriptions 
CTO Values and orientation of chemistry teachers towards the chemistry teaching process. 
KoC Chemistry teachers’ understanding of curriculum standards and textbooks. 
KoL Chemistry teachers’ understanding of students’ prior knowledge and learning difficulties. 
Kos Chemistry teachers’ understanding of chemistry teaching methods and representational means. 
KoA Chemistry teachers’ understanding of the dimensions and methods of chemistry teaching evaluation. 


Based on the contents and dimensions of the PCK components, the Chemistry PCK Questionnaire was ini- 
tially developed. The questionnaire consisted of 20 self-report items, utilizing a 7-point Likert scale. Higher scores 
indicated a higher degree of agreement (1=strongly disagree, 7=strongly agree). After the initial development 
of the questionnaire, a chemistry education expert assessed the relevant questionnaire items. The assessment 
criteria included the scientific nature of each component dimension, the accuracy and appropriateness of item 
design, and the correctness and standardization of item statements. The modified questionnaire, based on expert 
opinions, was used in the study. 


1) Reliability test 
After the data collection, the reliability analysis of the questionnaire was conducted using Cronbach's a 
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reliability coefficient in the pre-test and post-test phases. In the pre-test, the Cronbach's a values for each factor 
exceeded .91 (> .70), and in the post-test, the Cronbach's a values for each factor exceeded .87 (> .70), indicating 
good reliability of the questionnaire. 


2) Exploratory Factor Analysis (EFA) 

Exploratory factor analysis was first performed separately on the data from the 20 items in both the pre-test 
and post-test using SPSS 23.0 software, resulting in the identification of 5 factors. Specifically, the appropriateness 
of factor analysis was initially tested for all items. The results showed that in the pre-test, the KMO (Kaiser-Meyer- 
Olkin) value was .94 (> .80), and Bartlett’s sphericity test y? = 3838.36, p < .001. In the post-test, the KMO value was 
.94 (> .80), and Bartlett’s sphericity test y? = 3186.35, p < .001. These findings indicate that both pre-test and post- 
test data were suitable for factor analysis. The total explained variance for the 5 factors obtained in the pre-test 
was 82.14%, and in the post-test, it was 77.71%, both exceeding 70% (see Table 2 for the variance explained by 
each factor), indicating their strong explanatory power for PCK. Additionally, as shown in Table 2, the exploratory 
factor loadings for each item in the pre-test ranged from .66 to .87, and in the post-test, they ranged from .68 to 
.87, all exceeding .50, suggesting good initial validity for each item. 


Table 2 
Results of EFA 
Pre-test Post-test 
Component Item 
Factor loading Variance(%) Factor loading Variance (%) 
Students’ opinions are important and should be 82 (85) 87 (84) 
taken seriously 
Effective chemistry teaching should encourage 
more student discussion and hands-on practice 5189) eae) 
CTO Studying chemistry means students have plenty 17.31 16.17 
of opportunities to explore, discuss and express 87 (.88) 81 (.82) 
their ideas 
Agood chemistry classroom should have a demo- 
cratic and free atmosphere to promote students’ 82 (.80) TT (.80) 
thinking and communication 
am fam iar with the course objectives in the 79 (.84) 73(73) 
chemistry curriculum standards 
am familiar with the content standards required 
in the chemistry curriculum standards wee) oneal 
KoC am familiar with the distribution of modules in 17.16 14.46 
: .80 (.90) 75 (.77) 
chemistry textbooks 
am familiar with the connection and continuity 
of contents in chemistry textbooks (junior/high .72 (.84) 11 (.79) 
school, compulsory/elective) 
know the student's prior chemistry knowledge or 74 (87) 78 (83) 
learning experience 
| understand students’ prior core literacy in 17 (89) 83 (87) 
chemistry 
KoL ——— SS Se 15.02 a! 15.77 
| understand possible learning difficulties or 
inadequacies in students’ abilities cot) fe 
| understand possible misconceptions or misun- 73 (82) 73 (83) 


derstandings that students may have 
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Pre-test Post-test 
Component Item 
Factor loading Variance(%) Factor loading Variance (%) 


can use different teaching methods for different 


chemistry modules ta) Oats) 
| can use various activities to enhance students 17(89) 73 (85) 
nterest in learning chemistry 
KoS can use real-life examples to explain chemistry 16.83 15.38 
at .78 (.85) 80 (.85) 
concepts and/or principles 
can explain concepts and/or principles through 
demonstrations (e.g., physical objects, experi- .19 (.86) 80 (.84) 
ments) 
know how to evaluate students’ understanding 
and application of chemistry knowledge H0(a0) Hat) 
know how to assess student's learning abilities 75 (81) 68 (.89) 
in chemistry 
KoA 15.82 — 15.93 
| know how to evaluate students’ attitudes and 77 (84) 75 (.86) 
values toward chemistry 
| know how to assess students’ core competency 17 (84) 71 (83) 


levels in chemistry 


Note 1: Inside and outside the brackets are confirmatory factor loadings and exploratory factor loadings respectively. 
Note 2: CTO: Chemistry Teaching Orientation; KoC: Knowledge of Curriculum; KoL: Knowledge of Learners; KoS: Knowledge of 
Strategies; KoA: Knowledge of Assessment 


3) Confirmatory Factor Analysis (CFA) 

To further assess the structural validity, convergent validity, and discriminant validity of the questionnaire, 
CFA was conducted using AMOS 22.0. 

Model fit was first assessed to evaluate the adequacy of the structural model in this study. The fit indices for 
both the pre-test and post-test are reported below (Table 3). These results indicate that the model exhibited good 
structural validity in both the pre-test and post-test (Yaslioglu & Yaslioglu, 2020). The y?/df was below the recom- 
mended threshold of 3.00, suggesting a good fit between the model and the observed data. The RMSEA and SRMR 
values were also below the cutoff of .08, indicating a close fit of the model to the data. Additionally, the NFI, IF, TLI, 
and CFI values were all above the minimum threshold of .90, further supporting the model’s adequacy. The model 
fit results provide evidence that the proposed structural model adequately represents the relationships among 
the observed variables in the study. 


Table 3 
Model Fit 
Test Type df RMSEA SRMR NFI IFI TLI CFI 
Pre-Test 2.06 .07 .05 92 .96 95 95 
Post-Test 1.50 05 .04 93 97 97 97 


Note: y?/df. Chi-square/degrees of freedom; RMSEA: Root Mean Square Error of Approximation; SRMR: Standardized Root Mean 
Square Residual; NFI: Normed Fit Index; IFI: Incremental Fit Index; TLI: Tucker-Lewis Index; CFI: Comparative Fit Index 


Composite reliability and convergent validity were assessed to evaluate the reliability of the measurement 
tool in this study. As presented in Table 2, confirmatory factor loadings were calculated for each item in both the 
pre-test and post-test. The loadings ranged from .80 to .90 in the pre-test and from .73 to .89 in the post-test, all 
surpassing the threshold of .50. This indicates that each item adequately represents its respective factor. To fur- 
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ther examine the reliability of the factors, the average variance extracted (AVE) and construct reliability (CR) were 
computed for each factor. In the pre-test, AVE values ranged from .71 to .77, exceeding the recommended value of 
50. Additionally, CR values ranged from .91 to .93, surpassing the minimum threshold of .70. Similar results were 
obtained in the post-test, with AVE values ranging from .62 to .75 and CR values ranging from .87 to .92. The high 
factor loadings, along with the AVE and CR values exceeding the recommended thresholds, provide evidence for 
the robustness and consistency of the measurement tool used in this study (Cheung et al., 2023). 

Discriminant validity was assessed to examine the distinctiveness of the factors in the questionnaire. The 
correlation matrix presented in Table 4 shows that all correlations between factors in both the pre-test and post- 
test were statistically significant (p < .01). The correlation coefficients ranged from .39 to .76 in the pre-test and 
from .47 to .67 in the post-test. To evaluate discriminant validity, the square root of the average variance extracted 
(AVE) for each factor was compared to the Pearson correlation coefficients. The results revealed that all correlation 
coefficients were lower than the square root of the corresponding AVE in both the pre-test and post-test (ROnkk6 
& Cho, 2022). This indicates that while there was some interrelation between the factors, they also demonstrated 
sufficient discriminant validity. 


Table 4 
Results of Discriminant Validity 


KoC KoL KoS KoA CTO 
5 
KoC (62) 
+E Bee 15 
KoL .68** (.65**) (71) 
+e / Bee ae / BARK JT 
KoS .66** (.65**) .66** (.64**) (70) 
KoA 61** (.67**) 76** (.65*) 65** (.59*) oh 
: . : : : : (75) 
Led aR ak kk ak aK ee RR 13 
CTO 49** (.40**) 51** (.45*) 53** (.50**) 39** (.47**) (67) 
r= 86 .87 88 84 85 
VAVE (.79) (.84) (.84) (.87) (.82) 


Note 1: The inside and outside brackets are post-test and pre-test respectively, the diagonal is AVE, **p < 0.01. 
Note 2: CTO: Chemistry Teaching Orientation; KoC: Knowledge of Curriculum; KoL: Knowledge of Learners; KoS: Knowledge of 
Strategies; KoA: Knowledge of Assessment 


The results of the CFA mentioned above indicate that there are significant correlations among the first-order 
factors of PCK in both the pre-test and post-test phases (see Figure 3). Based on the PCK five-component model, 
a second-order structural model was established (see Figure 4), and its fit to the data was examined through 
higher-order confirmatory factor analysis. The model fit of the pre-test and the post-test can be seen in Table 5. 
These results suggest that the second-order structural model is a good fit for the data (Yaslioglu & Yaslioglu, 2020). 
Additionally, the factor loadings of the first-order factors in the pre-test ranged from .60 to .90 (> 0.5), and in the 
post-test, they ranged from .59 to .89 (> .5). This indicates that the lower-order factors have high explanatory power 
for the higher-order factor, which further supports the validity of the second-order structure. 


622 


https://doi.org/10.33225/jbse/24.23.615 


Journal of Baltic Science Education, Vol. 23, No. 4, 2024 


DEVELOPING CHEMISTRY PRESERVICE TEAGHERS’ PEDAGOGICAL CONTENT KNOWLEDGE ISSN 1648-3898 = /erinty 
(PGK) THROUGH THE LEARNING BY COLLABORATIVE DESIGN (LBCD) CURRICULUM MODEL 
(PP. 615-631) ISSN 2538-71 38 /Online/ 


Figure 3 
First-order Pairwise Correlation Structure of PCK 


Note 1: The left side is the pre-test, and the right side is the post-test. 
Note 2: CTO: Chemistry Teaching Orientation; KoC: Knowledge of Curriculum; KoL: Knowledge of Learners; KoS: Knowledge of 
Strategies; KoA: Knowledge of Assessment 


Figure 4 
Second-order Structure of PCK 


Note 1:The left side is the pre-test, and the right side is the post-test. 
Note 2: CTO: Chemistry Teaching Orientation; KoC: Knowledge of Curriculum; KoL: Knowledge of Learners; KoS: Knowledge of 
Strategies; KoA: Knowledge of Assessment 


Table 5 
Model fit 
Test Type /df RMSEA SRMR NFI IFI TLI CFI 
Pre-Test 2.14 07 .05 91 95 94 95 
Post-Test 1.53 .05 .05 92 97 97 97 


Note: y?/df: Chi-square/degrees of freedom; RMSEA: Root Mean Square Error of Approximation; SRMR: Standardized Root Mean 
Square Residual; NFI: Normed Fit Index; IFI: Incremental Fit Index; TLI: Tucker-Lewis Index; CFI: Comparative Fit Index 


Unstructured Interviews 


After the intervention, 30 participants were randomly selected for interviews to find out their views of the 
effectiveness of the intervention and the deep reasons behind these views. The interview questions mainly cov- 
ered topics such as “How did you complete the various sections of your teaching design before the course?” “How 
did you complete the various sections of your teaching design after the course?” “What were the reasons for the 
changes in your teaching design method before and after the course?”, and “In your opinion, what impact does 
group collaboration in teaching design have on your learning?” 


Data Analysis 


First, after the development of the chemistry PCK questionnaire, Exploratory Factor Analysis (EFA) and Con- 
firmatory Factor Analysis (CFA) were used to assess the quality of the questionnaire. EFA was employed to assess 
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the goodness of fit of the PCK five-component model (a first-order model with pairwise correlations). Based on the 
strong correlations among the first-order factors of the questionnaire, high-order CFA was conducted to assess the 
goodness of fit of the “second-order” structure of the questionnaire. Besides, Paired-Samples t-tests were also used 
to compare the levels of individual PCK components among preservice chemistry teachers before and after the 
intervention. Additionally, interview data were utilized to explain further changes in the levels of PCK components 
and the reasons behind these changes. This study employed SPSS 23.0 and AMOS 22.0 software for data analysis. 


Research Results 
Pre-service Teachers’ Chemistry PCK Performance 


Before and after the intervention, the normality test was performed on the pre and post-test results. Based on 
the data conforming to the normal distribution, descriptive statistics and paired sample t-tests were conducted on 
the scores of PCK in the pre-test and post-test to examine the effectiveness of the LBCD curriculum model in promot- 
ing the development of PCK components. The results (Table 6) showed that there are significant differences in the 
mean of PCK components, including CTO, KoC, KoL, KoS, and KoA, between the pre-test and post-test. Comparing 
the post-test to the pre-test, there were improvements in the mean scores of all PCK components. The degree of 
improvement, from highest to lowest, is ranked as follows: KoL (0.59), KoA (.53), KoC (.49), KoS (.45), and CTO (.44). 
This suggests that the LBCD curriculum model has a statistically significant positive effect on the development of 
PCK components, and the development of each component shows some degree of unevenness. 


Table 6 
Results of Descriptive Statistics and Paired Sample t-test 


Component Paired M SD t df p 

CTO pre-test 4.85 1.19 

CTO 417 209 <.01 
CTO post-test 5.29 .90 
KoC pre-test 4.56 1.35 

KoC -4.20 209 <.001 
KoC post-test 5.05 98 
KoL pre-test 4.34 1.30 

KoL -5.28 209 <.001 
KoL post-test 4.93 1.01 
KoS pre-test 5.02 1.43 

KoS -3.60 209 <.01 
KoS post-test 5.47 92 
KoA pre-test 4.29 1.20 

KoA -4.73 209 <.001 
KoA post-test 4.82 1.06 


Note: CTO: Chemistry Teaching Orientation; KoC: Knowledge of Curriculum; KoL: Knowledge of Learners; KoS: Knowledge of Strategies; 
KoA: Knowledge of Assessment 


Pre-service Teachers’ Perceptions of the Effectiveness of the LBCD Curriculum 


A detailed analysis of the improvement in the levels of PCK components based on individual questionnaire 
items was provided, and further explanation was based on the responses of preservice teachers to interview ques- 
tions (see Table 7). 
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Example of Interview Results 


Interview Questions 


1. How did you complete 


the writing of each part of 


the instructional design 
before studying the 
course? 


2. How did you complete 
the writing of each part 
of the instructional 
design after studying the 
course? 


3. What are the reasons 
for the change in your 
method of writing in- 
structional design before 
and after studying the 
course? 


4. What effect do you 
think working in groups 
on instructional design 
has on your learning? 


Sample interview responses for preservice teachers 


Student 5 Before class, my understanding of chemistry classes was based on the traditional model of teachers lecturing 
and students listening. My knowledge of curriculum standards and textbooks is limited to the outdated textbooks | used in 
high school. | have heard about curriculum standards, but | don’t fully understand them. When analyzing students’ learning, 
| mainly rely on my observations and personal experiences, or | refer to the literature. 

Student 11: For activity design, | learned some teaching methods during my sophomore year, such as question and answer 
methods, experimental methods, etc., as well as teaching tools like pictures and videos. However, when it comes to lesson 
planning and teacher activities, | usually rely on my intuition or use existing teaching materials. In terms of evaluation, | 
primarily assess students’ understanding of chemistry knowledge and help them reinforce and check their understanding 
through exercises and tests after class. 


Student 5:! have a better understanding of the new textbooks and curriculum standards when analyzing them. It’s important 
to examine the content and organization of the teaching materials and outline the topics to be covered in class. While cur- 
riculum standards are consistent nationwide, textbooks may vary. 

Student 13: | understand the significance of conducting a well-founded analysis of students’ learning. This analysis should 
summarize students’ existing knowledge, identify their learning difficulties, and evaluate their abilities, conceptual under- 
standing, and literacy. It should be based on references to textbooks, curriculum standards, and empirical research papers. 
However, it's important to avoid directly quoting literature conclusions and instead provide an analysis that considers the 
specific context. 

Student 11: In terms of activity design, | feel more comfortable with designing teaching situations, selecting teaching methods, 
and utilizing different teaching representation techniques. The design of learning tasks and activities should be logical and 
structured, incorporating various teaching representation methods effectively. It’s important to prioritize the scientific and 
appropriate use of these methods. 

Student 17: | have learned various evaluation tools and methods such as concept maps and evaluation rubrics. 


Student 3: In my case, the change occurred when | studied the instructional designs of other groups in class. Although all 
the groups delivered excellent presentations, my group presented later in the class. As we listened to the presentations of 
other groups, we made modifications to our designs. We learned from the strengths of other groups and took the opportunity 
to self-assess and correct any shortcomings. 

Student 16: The main reasons were the teacher's constructive feedback during each class and the comments from other 
groups. These suggestions also triggered our group's reflection and promoted our continuous revision. 

Student 28: This change came about through an iterative process of revisions, driven primarily by the teacher's guidance. 
After one of our group presentations, the teacher asked us about how we planned to implement the evaluation objectives. 
When our group was unable to provide a satisfactory response, it prompted us to reflect deeply on the selection and design 
of the evaluation tool. Following this feedback, our group refined the evaluation tool to better address this aspect. 


Student 9: Working side by side with the team members did help a lot. It was really difficult for me to persevere through 
revision after revision. 

Student 13: Our group was a productive community. Everyone was willing to help each other while discussing and modifying 
our design products. 

Student 22: When we compared our work with other groups, we often realized our shortcomings. This motivated us to further 
revise our work. The more we revise, the better our understanding becomes. 


CTO (Chemistry Teaching Orientation). Before and after the intervention, preservice chemistry teachers showed 


an improvement in the average scores for all items related to CTO. The items with the greatest improvement, in 
descending order, were “studying chemistry means students have plenty of opportunities to explore, discuss and 
express their ideas” (.64), “Effective chemistry teaching should encourage more student discussions and hands-on 
practice” (.42),“A good chemistry classroom should have a democratic and free atmosphere to promote student 
thinking and communication’ (.38), and “Student perspectives are very important and should be taken seriously” 
(.32). Based on the responses of preservice teachers to interview questions 1 and 2, Many preservice teachers men- 
tioned in the interviews that their impressions of classroom organization in chemistry before the intervention were 
based on a teacher-centric model where the teacher talks and students listen. After the intervention, preservice 
teachers had a deeper understanding that the classroom should be student-centered, providing students with 
more opportunities for exploration, discussion, and expression in chemistry classes. 

KoC (Knowledge of Curriculum). Before and after the intervention, preservice teachers also showed improve- 
ment in the average scores for all items related to KoC. The items with the greatest improvement were“! am familiar 
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with the content standards required in the chemistry curriculum standards” (.55) and“I am familiar with the connec- 
tion and continuity of contents in chemistry textbooks” (.55). On the other hand, the items with relatively smaller 
improvements were “I am familiar with the distribution of modules in chemistry textbooks” (.44) and “lam familiar 
with the course objectives in the chemistry curriculum standards” (.44). It was revealed through interviews that 
preservice teachers had not studied courses related to the analysis of chemistry curriculum standards or the analysis 
of middle school chemistry textbooks before the intervention, and their awareness of curriculum standards was 
limited, mostly based on their experiences in middle and high school. After the intervention, preservice teachers 
became more familiar with the distribution and connection of textbook content, as well as the course objectives 
and content standards in the curriculum. 

KoL (Knowledge of Learners). Before and after the intervention, preservice teachers improved the average 
scores for all items related to KoL. The items with the most significant improvement, in descending order, were “| 
understand possible misconceptions or misunderstandings that students may have’ (.70), “| understand possible 
learning difficulties or inadequacies in students’ abilities” (.59), “l understand students’ prior core competencies in 
chemistry” (.56), and “I know students’ prior chemistry knowledge or learning experience” (.52). According to the 
interviews, preservice teachers’ knowledge about student learning before the intervention was mainly based on 
their learning experiences, literature, and educational materials. After the intervention, preservice teachers recog- 
nized the need to analyze students’ knowledge and learning difficulties based on curriculum standards, textbooks, 
and empirical papers with specific data from various dimensions, including knowledge, skills, methodological 
abilities, and conceptual literacy. 

KoS (Knowledge of Strategies). Before and after the intervention, preservice teachers demonstrated improve- 
ment in the average scores for all items related to KoS. The items with the greatest improvement were “I can use 
real-life examples to explain chemistry concepts and/or principles” (.54) and “I can use various activities to enhance 
students’ interest in learning chemistry” (.49). On the other hand, the items with relatively smaller improvements 
were “I can use different teaching methods for different chemistry modules” (.39) and “I can explain concepts and/or 
principles through demonstrations (e.g., physical objects, experiments)” (.36). Many preservice teachers mentioned 
in the interviews that they had studied educational courses before the intervention, so they had some understand- 
ing of teaching methods and teaching representation methods, although their understanding was relatively vague. 
After the intervention, preservice teachers had a deeper understanding of the selection of chemistry teaching 
methods, the design of learning task activities, and the application of teaching content representation methods. 

KoA (Knowledge of Assessment). Before and after the intervention, preservice teachers showed improve- 
ment in the average scores for all items related to KoA. The items with the greatest improvement were “I know 
how to evaluate students’ attitudes and values towards chemistry” (.60) and “I know how to assess students’ core 
competency levels in chemistry” (.56). In comparison, the items with relatively smaller improvements were “I know 
how to assess student's learning abilities in chemistry” (.51) and “I know how to evaluate students’ understanding 
and application of chemistry knowledge” (.47). It was revealed through interviews that, influenced by the current 
emphasis on assessing students mainly in terms of knowledge and skills and the predominant use of paper-and- 
pencil tests as the primary assessment method in secondary school chemistry education, preservice teachers 
before the intervention had limited considerations for evaluating students from higher dimensions such as core 
competencies in chemistry, fundamental concepts, etc. After the intervention, preservice teachers began to de- 
velop an awareness of designing assessment objectives based on various dimensions, including knowledge, skills, 
methodological abilities, conceptual literacy, etc., and selecting or developing assessment tools. 

Besides, based on the responses of preservice teachers to interview questions 3 and 4, the facilitating effect 
of the LBCD curriculum model on the development of PCK components was explained (see Table 4). Regarding 
interview question 3, preservice teachers frequently mentioned keywords such as “repeated revisions,’“learning 
from each other,’ “reflection,’“mentoring feedback and guidance,’ and “capitalizing on strengths and addressing 
weaknesses.’ In response to interview question 4, preservice teachers often mentioned keywords like “common 
progress,’ “professional learning community,’ “mutual achievements,’ “collective brainstorming,’ “working side by 
side,’ and “collaborative teamwork.’ From this, it is evident that within the course, preservice teacher groups have 
formed a professional learning community. Within this community, they have transformed the individual“I"into the 
collective “we.’ Through continuous collaborative design, mutual reflection, joint problem-solving, and collabora- 
tive revisions, they have facilitated the development of collective PCK within the community. Collaborative group 
work in designing teaching has provided preservice teachers with a platform for dialogue and mutual assistance 
and has stimulated their subjective initiative. After actively absorbing collective PCK from the community, they 
gradually internalize it as their personal PCK, resulting in mutual achievements and collective progress. 
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Discussion 


This study developed a high-quality pre-service chemistry teacher PCK questionnaire using the SEM and 
Magnusson et al’s (1999) five-element model of teacher PCK. The study also implemented the LBCD teaching 
model to support the development of pre-service chemistry teacher PCK. 

Regarding research question 1, many previous studies have explored methods for assessing PCK. For example, 
Schubatzky et al. (2023) investigated the development of digital media teaching content knowledge in physics 
teacher education through workshop-based approaches, while Kutluca (2021) designed and conducted interviews to 
determine the PCK levels of elementary school teachers. However, these studies mostly used qualitative assessment 
methods and lacked effective quantitative assessment tools (Schiering et al., 2023). Currently, Magnusson’s five- 
element PCK model is the most widely applied (Leijen et al., 2022). Based on this model, the researchers designed 
a 20-item PCK scale for pre-service chemistry teachers. To validate the reliability and validity of this measurement 
tool, the researchers calculated Cronbach's a reliability coefficient and conducted EFA and CFA. By calculating 
Cronbach's a reliability coefficient, researchers can assess the internal consistency of the scale (Bland et al., 1997). 
In addition, EFA and CFA can verify the distribution of the five components of PCK and confirm whether the scale’s 
structure aligns with expectations (Organ, 2018). Through these validation steps, this study developed a quantita- 
tive assessment tool for pre-service chemistry teacher PCK, enabling a comprehensive understanding and analysis 
of PCK development. It also provides an effective measurement tool for future teacher training. 

For research question 2, developing chemistry teacher PCK, this study employed a quasi-experimental research 
method and designed and implemented the LBCD teaching curriculum model. This model aimed to engage pre- 
service chemistry teachers in collaborative learning experiences within a supportive practice community to enhance 
their understanding of subject matter content, teaching strategies, and instructional design (Stammes et al., 2020; 
Weinberg et al., 2021). The curriculum design consisted of five phases: collaborative design analysis, collaborative 
design guidance, collaborative design construction, collaborative design refinement, and collaborative design 
reflection. These five phases formed a continuous cycle of “collaborative design-reflection-revised collaborative 
design-revised reflection” to develop collective PCK among the pre-service teacher cohort and promote individual 
PCK development (Chai et al., 2020). Through carefully designed pre- and post-assessments using questionnaires, 
it was found that the LBCD curriculum significantly affected various dimensions of pre-service chemistry teacher 
PCK. The improvement levels ranked from highest to lowest were KoL, KoA, KoC, KoS, and CTO. 

Through an in-depth analysis of the reasons for the significant improvements in KoL and KoA, it was found 
that the current emphasis on knowledge and skill transmission and paper-and-pencil testing as the primary means 
of evaluating student development in secondary school chemistry education due to academic pressures limits 
pre-service chemistry teachers’ understanding of student learning. Their understanding is primarily based on their 
learning experiences, literature, and textbooks, with limited consideration of higher-dimensional aspects such 
as chemical literacy and fundamental concepts (Ye et al., 2021). However, after participating in the intervention 
curriculum, pre-service teachers realized the need to comprehensively understand student learning from various 
dimensions, such as knowledge, skills, methodological abilities, and conceptual understanding, based on curriculum 
standards, textbooks, and empirical papers (Vojif & Rusek, 2022). They began consciously designing assessment 
goals from different dimensions and selecting or developing assessment tools (Tomasevic et al., 2021). This shift 
enabled them to analyze students’ knowledge levels and learning difficulties more accurately and provide targeted 
support and guidance (Navy et al., 2021). The intervention curriculum may have provided theoretical and practical 
knowledge of multidimensional assessment, helping pre-service teachers better understand and apply assessment 
methods. They may have learned how to design challenging and inspiring tasks based on the requirements of 
subject literacy to assess students’ understanding of fundamental concepts, mastery of experimental skills, and 
ability to solve chemical problems (Babin¢éakova et al., 2020). 

Additionally, the intervention curriculum may have provided specific data analysis tools and techniques to 
help pre-service teachers better utilize student assessment data, identify student learning needs, and develop 
personalized teaching plans (Boesdorfer et al., 2022). On the other hand, teaching beliefs require time and accu- 
mulated practice to learn and understand, and CTO, as an essential dimension in the PCK model, often undergoes 
slow changes over time (Hashweh, 2005; Jay, 2023). Compared to other PCK dimensions, the development of a 
CTO may require longer-term interventions and continuous support (Ekiz-Kiran & Boz, 2020). 
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Conclusions and Implications 


Through designing the Chemistry PCK questionnaire and implementing the LBCD curriculum for preservice 
chemistry teachers, this study yielded three main conclusions: (1) The designed chemistry PCK questionnaire ex- 
hibited high quality, indicating its effectiveness in measuring PCK. (2) Implementing the LBCD curriculum model 
resulted in a statistically significant positive impact on developing PCK components among preservice chemistry 
teachers. However, it was observed that the growth across these components was uneven. The improvement levels 
across different dimensions of PCK varied, with the highest improvement seen in KoL, followed by KoA, KoC, KoS, 
and finally, CTO. Based on these findings, several recommendations can be made for chemistry PCK assessment and 
teacher education. The findings of this study highlight the effectiveness of the designed chemistry PCK question- 
naire in measuring PCK. Researchers and educators can employ this questionnaire to gain valuable insights into the 
pedagogical content knowledge of chemistry teachers and inform targeted professional development programs. 
Besides, in chemistry teacher education, the LBCD curriculum model developed in this study demonstrates innova- 
tive design and practicality in implementation, showing effectiveness in practice. It serves as a valuable reference 
for higher education of chemistry teachers. For instance, based on the success of the LBCD model, peer assistance, 
mentoring programs, and the establishment of professional learning communities for teachers can be considered. 

This study also has limitations. Regarding PCK assessment, future research could consider applying the 
questionnaire designed in this study to different groups of chemistry teachers. This would allow for quantitative 
exploration of PCK component levels among different groups and further exploration of the relationship between 
chemistry teacher PCK and factors such as subject knowledge, teaching experience, and teacher beliefs. It also 
provides an opportunity to validate the reliability and validity of the questionnaire further. Second, the question- 
naire developed in this study focused on assessing PCK in the domain of chemistry but needed more specificity in 
terms of subject matter. Subsequent research could enhance the thematic specificity of questionnaire develop- 
ment by designing PCK questionnaires tailored to specific chemistry topics. Besides, the LBCD curriculum model 
significantly impacted the development of PCK components among preservice chemistry teachers. However, the 
uneven growth observed across these components suggests the need for further exploration and refinement of the 
curriculum model. Educators and curriculum designers should consider tailoring the model to address preservice 
chemistry teachers’ specific needs and challenges, ensuring a more balanced development of PCK components. 
Lastly, this study lacked a control group due to the limitations of time and resources. Future research should em- 
ploy controlled experiments or comparative studies to determine the differences in PCK development between 
the LBCD curriculum intervention group and a blank control group. 
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Abstract. Knowing the factors affecting 
fear of large carnivores is important for 
wildlife management and conservation. 
In this regard, the effect of worldviews 

of human-wildlife and human-nature 
relationships on the fear of large carnivores 
needs to be investigated in different 
cultures. In this study, the influence 

of wildlife value orientations (WVOs), 
connectedness with nature, religiosity, 
fear of dogs, and gender on the fear of 
wolves and wild boars were examined. 
Quantitative research was conducted via 
a questionnaire on a convenience sample 
of 656 university students from state 
universities in various regions of Turkey. 
Domination orientation did not influence 
the fear of either animal. Mutualism 
negatively influenced the fear of wolves 
but did not significantly influence the fear 
of wild boar. Connectedness to nature 

is related negatively to the fear of both 
animals. More religious students feared 
wild boars more. Female students had a 
greater fear of both animals than male 
students. In addition, students with a fear 
of dogs had a greater fear of both animals. 
The relationship between worldviews 
and fear of large carnivores can vary 
across different species and needs to be 
investigated in different cultures. 
Keywords: wildlife value orientations, 
connectedness to nature, religiosity, fear of 
wolves, fear of wild boar 
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Introduction 


Wildlife evokes a wide variety of emotions in humans, such as admira- 
tion, disgust, and fear. Some of these are biological predispositions (Selig- 
man, 1971) that are products of evolutionary survival adaptations (Jacobs, 
2009). Emotions toward wildlife can also be acquired through learning, as 
in cultural transfers or conditioning via individual experiences (Jacobs et al., 
2012). An emotion often associated with large carnivores is fear (Ambarli, 
2016; Dervisoglu & Menzel, 2024b; Jacobs et al., 2014; Johansson & Karlsson, 
2011; Prokop et al., 2024; Roskaft et al., 2003). Fear of large carnivores can 
reduce the quality of life of people in carnivore territory (Skogen & Krange, 
2003) and also can reduce support for their protection (Hermann & Menzel, 
2013; Johansson et al., 2012; Prokop et al., 2024). Therefore, understanding 
the factors that explain the fear of large carnivores is an important issue for 
wildlife management and conservation. In this regard, Prokop et al. (2024) 
suggested that humans inherited fear of large animals from their mamma- 
lian ancestors who were exposed to the predation of large reptiles for long 
periods during prehistoric times. As environmental context, the perceived 
threat from large carnivores and the perceived inability to control their own 
behavior during encounters have been shown to play an important role in fear 
of the animal Johansson & Karlsson, 2011; Johansson et al., 2012). Previous 
negative experiences also seem to be related to being more afraid of large 
carnivores (Mohammadi et al., 2021; Valente et al., 2024). 

The fear of large carnivores can also be associated with sociodemo- 
graphic factors. For example, some studies have shown that people in wolf 
territory are less afraid of them (Ericsson & Heberlein, 2003; Raskaft et al., 
2003). On the other hand, people living in the territory of large predators 
could have more negative attitudes due to the risk of a dangerous encounter 
and damage (Roskaft, et al., 2007; Zimmermann et al., 2001). Indeed, find- 
ings covering larger time periods suggest that long-term coexistence with 
wolves is associated with more negative attitudes toward them in addition to 
agreater fear of them (Barmoen et al., 2024; Dressel et al., 2015; Ericsson et al., 
2010). Additionally, the fear of large carnivores can also vary among different 
interest groups. For example, it was found that livestock owners were more 
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afraid of wolves (Valente et al., 2024), and hunters were less afraid (Ericsson & Heberlein, 2003). Gender has also 
been reported to be associated with fear of large predators, with females generally being more afraid than males 
(Kaltenborn et al., 2006; Prokop & Fancovicova, 2010; Prokop et al., 2021; Raskaft et al., 2003; Siard & Jordan, 2018). 

The relationship between worldviews and emotions toward wildlife is poorly understood. For example, 
people can have more positive attitudes toward wildlife species that are local icons (Castillo-Huitrén et al., 2024). 
Superstitions and myths about wildlife can also influence emotions toward them. For example, Prokop et al. (2024) 
show that superstitious beliefs about large carnivores, e.g. that their powers and protection could be transferred 
to people wearing certain parts of them, indirectly influence admiration for them via admiration of non-animal 
objects. Similarly, it has been revealed that belief in myths about bats is associated with negative attitudes toward 
them (Musila et al., 2018; Prokop et al., 2009). Additionally, research (Dervisoglu & Menzel, 2024a; Landon et al., 
2020) has shown that individuals’ emotions toward wildlife can be influenced by their wildlife value orientations 
(WVOs), which reflect their worldview as it pertains to the human-wildlife relationship (Manfredo et al., 2009, 2020). 
People with mutualism orientation, which is one WVO, view wildlife as having similar rights as humans and as family 
members (social affiliation belief). They also find wildlife worthy of care and feel a positive emotional connection 
with them (caring belief) (Manfredo et al., 2009, 2020). According to domination, another WVO, wildlife should be 
used for the interest of humans (appropriate use belief) and hunting wildlife is acceptable (hunting belief). In one 
study (Landon et al., 2020) with Illinois residents, the “appropriate use belief” of domination was associated with 
stronger negative emotions toward wolves. “Hunting beliefs’, on the other hand, are associated with weaker nega- 
tive emotions. In this regard, studies have also shown that hunters (Ericsson & Heberlein, 2003) and people who are 
interested in hunting (Roskaft et al., 2003) are less afraid of wolves. Based on this, domination-oriented individuals 
are expected to be less afraid of large carnivores and also game animals. In a cross-cultural study (Dervisoglu & 
Menzel, 2024a) with a Turkish and German sample, German students with a stronger domination orientation were 
found to have stronger negative emotions (e.g. anger, sadness) toward problems caused by the presence of wolves. 
Mutualism, on the other hand, indirectly (negatively) affected these negative emotions by being associated with 
lower threat perceptions from wolves in the Turkish and German samples. Thus, mutualist-oriented individuals are 
also expected to have less fear of large carnivores. 

Religious beliefs can also shape people's views of nature (Slimak & Dietz, 2006) and wildlife (Bhatia et al., 2017; 
Li et al., 2014). Research findings on the relationship between religion and emotions toward wildlife are inconsis- 
tent. There is evidence that religious beliefs regarding wildlife can influence emotions felt toward it. For example, 
seeing elephants as gods relieves the stress felt about their potential to cause damage (Gogoi, 2018). Similarly, it 
has been shown that if there is no personal experience of a wolf attack, religious beliefs (e.g. a person being cursed 
if he/she kills a wolf) can influence a person's fear of wolves (Mohammadi et al., 2021). Religiosity was also found 
to be associated with a higher perceived threat from wolves and stronger negative emotions (e.g. anger) toward 
problems caused by the presence of wolves (Dervisoglu & Menzel, 2024a). On the other hand, Bhatia et al. (2017) 
report that religiosity was associated with positive attitudes toward snow leopards and wolves in Buddhist but 
not Muslim communities. In another study, Castillo-Huitrdn et al. (2024) found no significant relationship between 
religion and emotions toward wildlife. Religions generally encompass a wide range of beliefs about the world and 
also animals, so the relationship between religion and emotions toward wildlife needs to be examined in different 
religious groups and cultures. 

The biophilia hypothesis proposes that there is an evolutionary affinity between humans and nature (Kel- 
lert & Wilson, 1993). Accordingly, the extent to which people feel connected to nature may affect their emotions. 
Research points to a positive relationship between connectedness to nature and happiness (Capaldi et al., 2014). 
Likewise, Kubiatko et al. (2021) report that students with a fear of bees and wolves had a lower nature relatedness. 
Similarly, Cho and Lee (2018) observed that children who have more contact with nature are less afraid of wildlife. 
In addition, there is also evidence that people interested in nature activities (e.g. walking in nature) were less afraid 
of large carnivores than those who were not interested (Prokop & Fancovicova, 2024; Prokop et al., 2011; Roskaft 
et al., 2003). Therefore, people with a stronger connectedness with nature can be expected to be less afraid of 
large carnivores. 

Considering that value orientations and religious beliefs can vary among cultures, it is important to examine 
their influence on emotions toward wildlife in culturally diverse societies. The aim of this study is to examine the 
influence of WVOs, connectedness to nature, religiosity, and gender on the fear of wolves and wild boar in a sample 
from Turkey. Considering that some dogs resemble wolves, the fear of wolves can be a result of the fear of dogs. 
Therefore, the influence of the fear of dogs on the fear of wolves was also examined. Wolves (Canis lupus) and wild 
boar (Sus scrofa) are widespread and conflict-prone animals in Turkey, causing financial damage to farmers (Ambarli, 
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2019; Chynoweth et al., 2016). Both can cause injuries and fatalities to humans (Linnell et al., 2021; Mayer et al., 
2023) and both are animals that people are afraid of (Ambarli, 2016; Dervisoglu & Menzel, 2024b; Johansson et 
al., 2011; Roskaft et al., 2003). Wolves live in almost every region of Turkey (around 6,000 wolves) (Ambarli, 2019; 
Ambarli et al., 2016) and are protected species (Resmi Gazete, 2003). According to the Islamic faith, pigs (or wild 
boar) are forbidden to eat and are considered dirty (Foltz, 2014). This negative view of pigs in Islam could lead to 
more religious Muslims being more afraid of wild boar. Fear was examined as an emotional disposition (Jacobs et al., 
2012).We formulated the following hypotheses per the theoretical basis and empirical findings mentioned above: 


H,: Domination would negatively influence the fear of wolves and wild boar. 

H,: Mutualism would negatively influence the fear of wolves and wild boar. 

H,: Religiosity would positively influence the fear of wolves and wild boar. 

H,: Connectedness to nature would negatively influence the fear of wolves and wild boar. 
H,: The fear of dogs would positively influence the fear of wolves. 

H,: Female students would have a greater fear of wolves and wild boar than male students. 


Research Methodology 
General Background 


A subset of data from a larger study (Dervisoglu & Menzel, 2024a, 2024b) conducted in Turkey and Germany 
is presented here. A study was conducted using a paper-and-pencil questionnaire. The questionnaires were ad- 
ministered in April and May 2016 at some state universities in Eastern Anatolia, Central Anatolia, the Aegean, and 
the West Black Sea regions of Turkey. Participation was voluntary and students were informed in advance about 
the purpose of the study. 


Sample Selection 


A convenience sample of 656 university students (179 male, 469 female, eight unspecified) was used in the 
study. The students were between 18 and 36 years old (M = 21.71; SD = 2.19; 3.4% missing). The academic majors 
of participants were Chemistry/Chemistry Teaching (ca. 25%), Computer Teaching and Instructional Technologies 
(ca. 20%), Elementary Science Teaching (ca. 18%), Primary School Teaching (ca. 10%), Biology/Biology Teaching (ca. 
6%), Preschool Teaching (ca. 6%), other (ca. 6%), unspecified (ca. 9%). 71.2% of the participants reported aiming 
to become a schoolteacher (5.2% missing). The questionnaire asked about the participants’ religious affiliation. 
Overall, 93.6% of students said that they were Muslim, 0.3% said that they were Christian, 0.6% responded with 
other, and 5.3% said that they did not belong to a religion. 


Instruments 


The items measuring fear were part of a six-item scale measuring negative emotions toward problems caused 
by the presence of wildlife that was part of a questionnaire originally developed by Hermann and Menzel (2013, 
2015). The authors of this paper adapted the scales to the context of the “wildlife presence”. The questionnaire was 
originally in German and adapted to Turkish by back-translation. The German questionnaire was translated into 
Turkish by one of the authors, a bilingual expert in biology education, and then back-translated by another bilin- 
gual expert in biology didactics. All discrepancies that arose were discussed by both experts and they generated 
a consensus. For more detailed information about the whole questionnaire, see Dervisoglu and Menzel (2024a, 
2024b). To measure fear, participants were asked what they thought about the presence of wolves and wild boars. 
Two items were used:“When | think about the animal's presence, | already have a queasy feeling” and“I am afraid of 
the animal” A photo of a wolf, a wild boar, and also a short informational text about both animals were presented 
on the questionnaire. The items toward emotions included the word “animal’, referring to wolves and wild boar. 
Accordingly, each item had to be answered twice, once for the wolf and once for the wild boar. The questionnaire 
included the 19-item WVO scale (Manfredo et al., 2009) including some modifications (Hermann & Menzel, 2013). 
Both the emotion and WVO scales were 5-point (1: strongly disagree to 5: strongly agree). Connectedness to nature 
was measured with the scale “inclusion with nature in self (INS)” developed by Schultz (2002). This one-item scale 
consists of seven circles, one named “I” and the other “Nature”, overlapping to varying degrees. The extent of the 
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overlap between these two circles shows the connection with nature. Accordingly, participants were required to 
select the image that best described their relationship with nature. Religiosity was measured using three questions 
with four response options. Based on Levin et al. (1995) and Taylor et al. (1999), two questions measured subjective 
religiosity and one question measured performance of religious activities (Levin et al., 1995). To measure fear of 
dogs, the question was asked: “Are you afraid of dogs?” (yes/no). The binary variables for gender (with women as 
the reference group) and the fear of dogs were dummy coded before analysis. 


Data Analysis 


Missing values were completed by the mean of relevant items (excluding categorical variables). Since the data 
did not show a multivariate normal distribution, the bootstrap method (Byrne, 2001) was used (10,000 resamples). 
Confirmatory factor analysis (CFA) was performed to test the validity of the scales. We tested the reliability of the 
latent variables (WVOs, the fear of wolves, and wild boar, religiosity) by calculating Cronbach's alpha values. Notably, 
reliability could not be calculated for the INS since it only had one item. The hypotheses of the research were tested 
using structural equation modeling (SEM). Separate CFA and SEM models were created for both wolves and wild 
boar. The acceptability of the CFA and SEM models was examined with fit indices. Accordingly, the comparative 
fit index (CFI) > .90 and the root mean square error of approximation (RMSEA) < .08 indicated an acceptable fit 
(Bryne, 2001). In the SEM models, the direct effect of the variables on fear was examined. Since the same model 
was created for both animal species, fear of dogs was also included in the model for wild boar. Cases (17 in total) 
with missing values for gender and fear of dogs were removed before the SEM analysis. IBM-SPSS® and AMOS 
(version 23) were used to analyze the data. 


Research Results 
Results of Confirmatory Factor Analyses 


As mentioned, two separate measurement models were developed for wolves and wild boar. First, we con- 
ducted CFA for both animals separately. Two items (“Hunting does not respect the lives of animals” and “Hunting 
is cruel and inhumane to the animals”) were removed from the domination scale because of low factor loading, 
and a high standardized residual. By considering modification indices, theoretically justified error correlations 
(four in total) within the same latent constructs in the WVOs were added. Finally, the fit indices of the models with 
only latent variables (WVOs, religiosity, fear) were acceptable for both wolves (y?(df) = 572.888(199), p < .001, CFI 
= .927, RMSEA = .054; Bollen-Stine bootstrap p [BSp] < .001) and wild boar (y7(df) = 569.284(199), p < .001, CFI = 
.928, RMSEA = .053; BSp < .001). Standardized factor loadings of the scales were found to be above .40. Cronbach's 
alpha value was .82 for domination, .85 for mutualism, .84 for religiosity, .73 for fear of wolves, and .75 for fear of 
wild boar. The corrected item-total correlations of all items were > .40. The latent variables showed correlations 
with each other less than .90. Thus, discriminant validity was confirmed (Kline, 2011). 

According to Table 1, students were generally afraid of wolves and wild boar. In addition, the sample was 
highly religious. Mutualistic orientations were, on average, higher than domination orientations. 


Table 1 
Descriptive Information 


Construct N Min. Max. M SD Mdn 


Domination 656 1.00 4.63 2.65 0.73 2.63 
Mutualism 656 1.33 5.00 3.55 0.69 3.56 
Religiosity 656 1.00 4.00 2.95 0.70 3.00 
Fear of wolves 656 1.00 5.00 2.98 1.15 3.00 
Fear of wild boar 656 1.00 5.00 3.36 1.19 3.50 
Inclusion with nature in self 656 1.00 7.00 4.37 1.72 4.00 
Fear of dogs 647 0 1 0.35 0.48 

Gender (men) 648 0 1 0.28 0.45 
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After the validity and reliability of the latent variables were ensured, measurement models, including gender, 
INS, and fear of dogs, were created. The models showed an acceptable fit for wolves (y7(df) = 736.568(253), p < .001, 
CFI =.916, RMSEA =.055, BSp < 0.001) and wild boar (y7(df) = 758.410(253), p < .001, CFI =.912, RMSEA = .056, BSp < 
.001). Table 2 shows the correlations between fear and the other variables. There was a strong correlation between 
the fear of dogs and the fear of wolves and wild boar. There was also a stronger correlation between religiosity and 
fear of wild boar than fear of wolves. In addition, mutualism, INS, and the male gender were negatively correlated 
with the fear of both animals. 


Table 2 
Amos Correlations Between Fear and the Predictors 


Inclusion with 


Gender Mutualism Domination : Religiosity Fear of dogs 
nature in self 
Fear of wolves -.39"* --41* 09 -.38"* 20 52** 
Fear of wild boar -.31" -.32" 12 -.32" .36** 46** 


Bootstrap p values are presented (*: p <.01, **: p<.001) 


Results of Structural Equation Models 


Although domination was not significantly correlated with the fear of both animals, it was included in the 
model to control for its effect. The fit of SEM was acceptable for wolves (y?(df) = 736.568(253), p < .001, CFI = .916, 
RMSEA = .055; BSp < .001) and wild boar (7?(df) = 758.410(253), p < .001, CFI = .912; RMSEA = .056; BSp < .001). Fear 
of dogs positively predicted fear of wolves (8 = .33, p < .001) and fear of wild boar (8 = .28 p < .001). Mutualism (8 
=-.22, p< .01) negatively predicted fear of wolves, but it was not a significant predictor of fear of wild boar. On the 
other hand, domination had no predictive power over the fear of either animal. INS negatively predicted fear of 
wolves (8 = -.15, p < .001) and fear of wild boar (8 = -.16, p < .001), while religiosity was a positive predictor of fear 
of wild boar (8 = .20, p < .01), although it did not predict fear of wolves. Finally, gender negatively predicted fear 
of wolves (B = -.31, p < .001) and fear of wild boars (8 = -.21, p < .001), saying that females were more afraid. The 
explained variance was 46% for fear of wolves and 36% for fear of wild boar. 


Figure 1 
The Structural Equation Modeling Results for Wolves and Wild Boar 
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Note. Standardized values (8) are presented. Bootstrap p values are presented (*: p<.01, **: p<.001). The covariances are not 
shown so that complexity is reduced. 
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Discussion 


This research has shown that the influence of WVOs on fear of wildlife can vary by species. Domination was 
not associated with fear of either animal. Thus, H, was not supported. Considering that hunting beliefs and ap- 
propriate use beliefs can have opposing effects on negative emotions toward wolves (Landon et al., 2020), they 
may have neutralized each other’s effects. Therefore, the effects of the appropriate use and hunting dimensions of 
domination on fear should be examined separately in the future. On the other hand, consistent with the theory of 
WVOs (Manfredo et al., 2009), students with a mutualistic orientation are less afraid of wolves. Mutualism negatively 
correlated with both fear of wolves and wild boars, but this association was significant for wolves. Therefore, H, 
was only supported for wolves. However, the finding that a mutualistic orientation does not predict the fear of wild 
boar may be explained by the influence of religion. As we mentioned above, there are prohibitions and a negative 
view toward pigs in the Islamic faith. Therefore, the mutualistic beliefs of Muslim students may not pertain to this 
animal. Seeing wild boar as evil and even dirty animals can induce fear and even increase the perception of their 
danger; it has been claimed that disgust plays a role in the fear of animals (Armfield, 2006). The beliefs that wild 
boars are dangerous, dirty, and associated with sin may influence each other synergistically. Supporting this claim, 
religiosity predicted the fear of wild boar but not the fear of wolves, so H, was supported only for wild boar. The 
corresponding finding regarding wolves is consistent with studies that reported no relationship between religios- 
ity and negative attitudes toward particular wildlife species (Castillo-Huitron et al., 2024). In addition, Bhatia et al. 
(2017) also found no significant relation between religiosity and attitudes toward wolves in Muslim communities. 
Regarding fear of wild boar, the responses to the item “When | think about the animal's presence, | already have 
a queasy feeling’, in particular, may have reflected not only fear of but also a negative attitude toward wild boar 
due to the association of the animal with sin and uncleanliness. The findings from the same sample with this study 
showed widespread disgust and anger among students toward the wild boar (Dervisoglu & Menzel, 2024b). Nega- 
tive beliefs about wildlife species stemming from religion or myths can influence negative attitudes toward them, 
as shown by Musila et al’s (2018) and Prokop et al’s (2009) studies regarding bats. 

Supporting H,, the more students felt connected to nature, the less fear they felt toward wolves and wild 
boar. This indicates an emotional schema that includes cognitions like thought and memory: Students who are 
strongly connected with nature may have a more positive image and memory cognitions of wolves or wild boars. 
Therefore, biophilia (Kellert & Wilson, 1993) may serve as a filter for negative emotions toward wildlife. Alterna- 
tively, these people may have more knowledge about animals because they consume more books, magazines, or 
documentaries about nature since knowledge has been found to reduce the fear of large carnivores (Randler et 
al., 2020; Orazem et al., 2021). 

This study showed that students with a fear of dogs are more afraid of wolves than those without a fear of 
dogs, so H, was supported. On the other hand, the finding that students who are afraid of dogs are also more afraid 
of wild boars may be explained by a more general fear of large mammals, which is suggested to have evolutionary 
origins (Prokop et al., 2021, 2024). 

We found that female students are more afraid of both wolves and wild boar than male students, so H, was 
supported for both animals. This is usually explained as an evolutionary adaptation that arose because women 
were more vulnerable to lethal attacks from wild animals (Prokop & Fancovicova, 2013; Raskaft et al., 2003). In this 
regard, Prokop and Fancéovicova (2013) report that women have higher levels of disgust, fear, and danger percep- 
tions of animals which pose a threat of physical harm or illness. Some of the reported fears of wild boar may also 
be related to disease transmission, as the animal is seen as dirty due to the influence of Islamic belief. 

In this study, students were more afraid of wild boars than wolves. However, the data from the same sample 
in this study showed that students reported a stronger fear of wolves than wild boars when they imagined the 
animal (Dervisoglu & Menzel, 2024b). One explanation could be that in the emotion scale used in this paper, the 
students were asked about the emotions they felt due to the presence of animals, and wolves are not frequently 
present in the vicinity of humans in Turkey. 


Conclusions and Implications 
This research shows that the fear of wolves is influenced by worldviews pertaining to human-wildlife rela- 


tionships. Specifically, mutualistic beliefs seem to reduce the fear of wolves, but not of wild boar. The relationship 
between WVOs and the fear of various wildlife species should be examined in more detail in the future. Different 
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objects of fear should also be considered (Frank et al., 2015). For example, different results may occur in terms of 
fear of attack on humans and fear of attack on livestock. 

Research shows that the relationship between religion, emotions, and attitudes toward wildlife is quite complex. 
This study indicates that examining religion-based specific beliefs about wildlife species will yield more accurate 
results, rather than the general effect of religiosity or religiose affiliations on the emotions toward specific wildlife 
species. This is because religions include a variety of beliefs. Hence, in future, those elements of Islam which trig- 
ger fear of wild boar should be examined in more detail in Turkey and other Muslim societies, particularly through 
qualitative research. 

This study indicates that discourses and messages aimed at encouraging a mutualistic orientation are effec- 
tive in reducing the fear of large carnivores. Furthermore, strengthening people's connection with nature could 
be effective in reducing fear of large carnivores. 
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Abstract. The Many-Facet Rasch model is 
frequently used to analyse and minimize 
disparities in rater (judge) severity in 
performance evaluations, in which raters 
assign scores to test-takers’ performances. 
In this research, the aim of the present 
study was to analyse science teacher 
candidates’ laboratory activities by using 
the Many-facet Rasch model. Rasch model's 
surfaces are, respectively: 9 juries, 8 science 
activities and 24 criteria. The FACETS 
program was used to do data analysis. 
Findings show that laboratory activities, 
which were coded as E8, were found to be 
the most successful and E2coded activity 
was found to be the least successful based 
on the criteria. Jury numbered 7 or coded 
as G7, is the most lenient, and scorer 
numbered 5, or coded as G5, is the severest 
when the juries are listed from the most 
lenient. The study’s objective is to use the 
Many-Facet Rasch measurement model 

to analyse laboratory experiments linked 
to science-related activities. Analysis of 

the performance of science activities, 
analysis of criteria hardness, analysis of the 
severity and leniency of juries, and study 
of jury bias were carried out concurrently 
with this goal. At the end of the study, it 
can be easily inferred that the Multi-Facet 
Rasch measurement model could be used 
effectively to evaluate peer groups in 
science education and objective results 
could be obtained. 
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Introduction 


Science education aims to ensure that every individual participating in 
society has various basic and complex cognitive characteristics, conscious- 
ness, attitudes and values. In this regard, ensuring that individuals participate 
in society as individuals who can question, research, make inferences based 
on scientific foundations and make decisions in line with these inferences, 
and use, apply and develop science and technology is related to the quality 
of education in schools (Abd-El Khalick, et al., 1998). The trained workforce 
that countries need can only be provided by individuals graduating from 
the education system having the necessary skills. The development levels, 
economic situations and social welfare levels of countries are directly related 
to the skills of individuals participating in the workforce (Organization for 
Economic Co-operation and Development [OECD], 2019). All individuals need 
to acquire science process skills to use a rational perspective in their deci- 
sions, benefit from data and facts, and evaluate events and situations from a 
scientific perspective. In addition, having science process skills makes it easier 
for individuals to turn to science, technology and science-related professions, 
providing a high-value-added workforce to the country’s economy (Hamarat 
& Arkan, 2018; OECD, 2019). For this reason, measuring the science process 
skills of individuals, and completing and improving them can be seen as im- 
portant for both the individual and the country’s future. Therefore, scientific 
process skills could be acquired by individuals during their educational lives. 
To achieve this, teachers should understand the nature of science to know 
how to reach scientific knowledge, just like a scientist. Because the teacher 
interprets and transfers his/her own knowledge and experience to the student 
during the lesson (Hofstein & Lunetta, 1982). 

However, examining the level at which these skills are acquired is the 
most important step in the educational process. In order to ascertain the 
degree of science process skills attained by science teacher candidates, it 
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was crucial to employ one of the most significant phases of the measurement and evaluation teaching process in 
science education. Measuring science process skills, which had an important place in science education, would 
help to reveal potential deficiencies or strengths in pre-service teachers so that the necessary feedback correction 
studies could be carried out and future planning of teacher candidates could be made more accurately. When the 
studies containing all these classifications were examined, it was seen that pre-service teachers could acquire all 
skills through certain science activities (Harlen, 1999; Huppert, et al., 2002). Therefore, laboratory activities could 
help pre-service teachers learn scientific processes and give opportunities for the development of these skills by 
using these processes effectively (Duru et al., 2011). In particular, science teachers need to be familiar with scientific 
language and understand scientific processes in depth in order to teach science (Cotabish et al., 2011). Numerous 
investigations, however, have revealed that teachers lacked a theoretical understanding of these skills (Turkmen & 
Kandemir, 2011) and that science teacher candidates were not competent in teaching science process skills (Yilmaz- 
Tiiztin & Ozgelen, 2012). The majority of science teachers did not use skills such as identifying materials, observing, 
measuring and recording, collecting and interpreting experimental data in their lessons (Yandila & Komane, 2004). 
It showed that science teacher candidates’ knowledge level about science process skills was insufficient (Farsakoglu, 
et al., 2008). These findings demonstrated the necessity for teacher candidates—the future educators—to develop 
their science process skills. Three arguments were presented by Rowland et al. 1987 supporting the need for sci- 
entific education to use a laboratory approach centred on science process skills. The process approach, to start, 
placed a strong emphasis on science as a means of comprehending the natural world. Secondly, this approach 
would allow teacher candidates to better understand science subjects and the path followed by a scientist in the 
process of discovering events, principles and relationships in natural life. Thirdly, this approach would contribute 
to the development of scientific attitudes as it ensured the active participation of teacher candidates. 

The scientific curriculum includes science process skills. The skills that scientists employed in their research 
included observing, measuring, categorizing, collecting data, formulating hypotheses, utilizing data to build models, 
modifying and controlling variables, and doing experiments (MONE, 2018, p. 9). There was no clear approach to 
measuring and evaluating skills in the curriculum. A concentration on cognitive measurement of achievements rather 
than the measurement of science process skills in classroom assessment and evaluation processes resulted from 
the achievements’ obvious lack of science process skills (Duruk et al., 2017; Wellington, 1989; Wu, 1994). Although 
measuring achievements was important in evaluating the quality of education, not measuring the science process 
skills contained in the achievements in detail caused the skill deficiencies of teacher candidates to remain hidden 
(Anderson & Krathwohl, 2001). As a result, it became difficult for professors, parents or institutions to carry out the 
necessary follow-up, evaluation and development activities regarding the science process skill levels of teacher 
candidates, even in general. Science process skills were processes that were expected to be used and put to work 
regardless of the classification method. Upon reviewing the literature, it was found that many researchers have 
limited their assessment of science process skills to multiple choice questions (Aydogdu, 2017; Bahsi & Acikgil Firat, 
2020; Ergiil et al., 2011; Fathonah et al., 2018; Gill, 2019; Ozgelen, 2012; Sensoy & Yildirim, 2017; Uysal & Cebesoy, 
2022). Given the nature of science process abilities, information gathered through multiple choice questions alone 
would not be adequate. Nevertheless, other research (Aktamis & Ergin, 2007; Aktamis & Sahin Pekmez, 2011; Azizah 
etal., 2018; Indri et al., 2020; Rillero, 1998; Serevina et al., 2018; Strong, 2013) also benefited from using open-ended 
questions and activities. When the research’s limitations were taken into account, the primary issues were that all 
of these studies had a small sample size because they were conducted manually and face-to-face with participants 
(Leat & Nichols, 2000). Additionally, the study was not sustainable and there was insufficient data to comment on 
how the participants’ science process skills changed over time (Bearman et al., 2020; Lederman & Stefanich, 2007; 
Paine, 2022; Richardson & Clesham, 2021; Webb, 2007). To address these issues, it would be suitable to use peer and 
self-evaluation within the parameters of the study and to make use of the Many-Facet Rasch Model to guarantee 
objectivity in the measurement and evaluation process. 

Currently, a pivotal inquiry is: what methods exist for the objective evaluation of Laboratory Experiments 
grounded in Science Process Skills? This inquiry is central to the present research. A potential solution is found in 
the use of the Many-Facet Rasch Model (MFRM), which is rooted in Item Response Theory, as indicated by Semerci's 
series of studies from 2011 to 2012 and further supported by Yuztiak, Erten, Kara, and Kaptan’s research between 
2015 and 2019. MFRM yields two robust metrics: the separation and reliability indices. The reliability index can be 
equated to Cronbach's Alpha or KR-20, signifying the proportion of ‘True Variance’ in relation to‘Observed Variance’ 
High reliability scores, approaching 1.0, are preferred for both individuals and items, as highlighted by Linacre in 
2010. An expansion of Rasch measurement models is the Many Facet Rasch Model (MFRM) (Rasch, 1980; Wright & 
Stone, 1979). The Many-facet Rasch (Fk) equation looks like this: 
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Pnijk 
Praja Pe OG A 

The equation reads as follows: Bn is the examinee's ability; Di is the item’s difficulty; Cj is the severity of judge 
j; Fk is the additional difficulty overcome in being observed at the level of category k, relative to category k-1; and 
Pnijk is the probability of examinee n being awarded on item i by judge j a rating of k-1 (Linacre, 1989). Judges’ 
ratings of examinee performances are often necessary for authentic measurement (Linacre, 1994). The secret to 
this is MFRM. The purpose of this study is to assess how well MFRM can be used to prepare laboratory experiments 
based on science process skills. The preparation of experiments based on scientific process skills does not include 
closed-ended questions. For this reason, the most important tool used in scoring high-level skills is the rubric. Rat- 
ers make their grades according to the performance criteria in the scoring keys (Kan, 2007). However, the rater's 
decision is not based solely on performance. Different factors/surfaces affect scoring. These include difficulty 
level of the task/performance, rater strictness/generosity, ratee’s past, etc. These factors, which are not included in 
the content of the measurement, may influence the validity of the scoring (Prieto & Nieto, 2014). In this context, 
when scoring scientific process skills, the scores are: Rater effect; strictness/generosity, halo effect, avoidance of 
outliers; The difficulty of selecting criteria for performance; Rating criteria (points on the rating scale have different 
meanings among raters) affect variables. The rater effect is not related to the performance of the individual being 
scored but is one of the characteristic features of the rater. This variable interferes with measurements as an error 
and threatens the validity of measurement results (Eckes, 2005). 


Research Problem and Focus 


With the multi-facet Rasch model (MRM), rater characteristics can be analysed as a facet in the measurement 
model (Kose, et al., 2016). The preparation of experiments based on scientific process skills does not include closed- 
ended questions. For this reason, the most important tool used in scoring high-level skills is the rubric. Raters make 
their grades according to the performance criteria in the scoring keys (Kan, 2007). However, the rater’s decision 
is not based solely on performance. Different factors/surfaces affect scoring. These include difficulty level of the 
task/performance, rater strictness/generosity, ratee’s past, etc. These elements could affect the validity of the score 
because they are not part of the measurement’s substance (Prieto & Nieto, 2014). In this context, when scoring 
scientific process skills, the scores are: Rater effect; strictness/generosity, halo effect, avoidance of outliers; the 
difficulty of selecting criteria for performance; Rating criteria (points on the rating scale have different meanings 
among raters) affect variables. The rater effect is not related to the performance of the individual being scored 
but is one of the characteristic features of the rater. This variable interferes with measurements as an error and 
threatens the validity of measurement results (Eckes, 2005). 


Research Aim and Research Questions 


In this research, based on the multi-facet Rasch model (MRM), rater characteristics can be analysed as a facet 
of the measurement model. In parallel with this aim, not only the laboratory experiment performance, severity/ 
leniency, criterion hardness of science teacher candidates and their bias but also the opinions of pre-service sci- 
ence teachers are obtained to be analysed. 

The aim of this research was to use the Many-Facet Rasch measurement model to analyse laboratory experi- 
ments linked to science-related activities. For this purpose, performance analysis of science activities, criterion 
hardness analysis, jury severity/leniency analysis, and jury bias analysis were carried out. Throughout the study, 
the following research questions were tried to be answered: 

1) Howdothe rater characteristics of pre-service teachers analyse the laboratory experiments depending 
on the science-related activities? 

2) What are the opinions of the pre-service science teachers about using a Many-Facet Rasch measure- 
ment approach to analyse the prepared science laboratory activities based on science process skills? 
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Research Methodology 
General Background 


The book “Some Probabilistic Models for Intelligence and Attainment Tests” by George Rasch (1960) has a 
description of the Rasch model, a one-parameter logistic model based on item response theory. The Rasch model 
and the Many-facet Rasch model approach have been applied in the following fields: language testing (Eckes, 
2005); educational and psychological measurement (Ahmad, Ali & Zainudin, 2011; Chang & Engelhard, 2016; Cetin 
& Ilhan, 2017; Ismail, et al., 2017; Kaya, et al., 2017; Kése, et al., 2017; Semerci, 2012; Semerci, 2011; Yilmaz & S6zer, 
2018), and health sciences (Park, et al., 2018). 

Arelated study was carried out with pre-service science teachers in the 2023-2024 academic year. In this study, 
the survey research strategy was applied. By the ethical research rules (Aydin Adnan Menderes University Protocol 
Number = E-84982664-050.04-506435, Decision Date: 29.02.2024 Meeting Number: 2024/2-XIX), the laboratory 
experiments were coded as E1, E2 ... E8; criteria were coded as a qualitative problem, well-explained problem, 
well-content problem etc., and the student groups (jury) were coded as G1, G2, ... G8; expert was coded as G9. 


Participants 


In this research, participants consist of 27 pre-service science teachers who are the 3rd-grade students of 
the Science Laboratory Applications Course in the Department of Science Teaching in a university located in the 
western region of Turkey. Convenience sampling was used for the research due to its ease of use for both sample 
and study execution. Pre-service science teachers first evaluated themselves by making self-evaluation and then 
other groups based on the rubric given to them, individually. Later, the faculty member who took the course 
made evaluations within the framework of these competencies. With the help of this model, the best component 
of scientific process skill, which group has the better scientific process skill than the others and the objectivity of 
the evaluation process have been tried to be revealed at the end of the study. 


Instrument and Procedures 


The quantitative data from the research were analysed by using the Many-facet Rasch model. The criteria 
included 24 items that refer to measuring the degree of ability and skills in laboratory applications in terms of 
considering and evaluating the efficacy of laboratory works as science teacher candidates. Criteria and abbre- 
viations are indicated in Table 1 (Kaygisiz et al., 2017). For validity and reliability issues, based on Item Response 
Theory, the Many-Facet Rasch Model (MFRM) presents a viable answer, as demonstrated by Semerci’s 2011-2012 
set of studies and further reinforced by Yuztiak, Erten, Kara, and Kaptan’s 2015-2019 study. The separation and 
dependability indices are two strong indicators that MFRM produces. The dependability index, which represents 
the ratio of “True Variance” to “Observed Variance,’ is equivalent to Cronbach's Alpha, or KR-20. As Linacre (2010) 
pointed out, both persons and objects should have high dependability scores, close to 1.0. The Many Facet Rasch 
Model (MFRM) is an extension of Rasch measurement models (Rasch, 1980; Wright & Stone, 1979). This is how the 
many-facet Rasch (Fk) equation appears: 


Table 1 
Criteria and Related Abbreviations 


C Nu Criteria Criteria Abbreviations 
C1 The problem is worth for designing experiments to find a solution. Qualitative problem 
C2 The specified problem sentence is correct as a statement. Well-explained problem 
C3 The specified problem sentence is correct in terms of content knowledge. Well-content problem 
C4 The specified problem statement is solvable. Solvable problem 
C5 The hypothesis statement is aimed at solving the problem. Hypothesis and problem are parallel 
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C Nu Criteria Criteria Abbreviations 
C6 The hypothesis sentence is correct as a statement. Well-explained hypothesis 
C7 The dependent variable was determined correctly. Correct dependent variable 
C8 The independent variable was determined correctly. Correct independent variable 
ie] Control variables were determined correctly. Correct controlled variable 
C10 The design of the experiment is sufficient to solve the problem. Design is for problem solving 
C11 It was mentioned how to change the independent variable. Changing independent variable 
C12 It was mentioned how to measure the dependent variable. Measuring dependent variable 
C13 The materials in the experiment are suitable for the problem and design of the experiment. Convenient materials 
C14 The procedure steps are given in a logical integrity/sequence. Step by step movement 
C15 At least three tests were made to ensure the reliability of the experimental data. Reliability tested (at least 3 times) 
C16 Variables that needed to be controlled in the experiment were controlled. Correct controlled variables 
C17 Tables/graphs appropriate to the data related to the experiment were used. Convenient tables/graphics 
C18 Dependent and independent variable names are included in the table. Tabled variables 
C19 The units of the independent and dependent variables are written correctly. Correct units of variables 
C20 The measurements/findings in the experiment were calculated correctly. Correct measurement calculation 
C21 The data has been correctly placed in the table. Correct tables with variables 
C22 In the results, an explanation for the problem was made. Results explain the problem 
C23 As a result of the experiment, correct explanation was made in terms of content knowledge. — Well-explained content results 
C24 The hypothesis was taken into account when evaluating the experimental results. Hypothesis based evaluation 


Data Analysis 


Within the context of the Science Laboratory Applications-Il course, the research group's science teacher 
candidates were theoretically instructed on the content and components of science process skills. Some activities 
were presented to teacher candidates as an example. In the course implementation, closed-ended, semi-open- 
ended and open-ended experiment activities were carried out with the teacher candidates throughout the one 
semester. In addition to these, open-ended experiments were implemented to the science teacher candidates 
within the scope of the Science Laboratory Application-Il course. They were given a new scenario for each week, 
and they had to come to class by designing their experiments until the next class day. They have conducted their 
experiments during the class time, and they have taken notes of their data. All tasks were completed by science 
teacher candidates in groups, and they prepared an experiment report for every lab activity. After that, they have 
to come to class with their experiment reports as prepared for the following week. 

In the course, firstly, the reports of each experiment carried out the previous week were evaluated together 
with all groups in the classroom and the course instructor, based on the rating scales, which were shared with the 
teacher candidates. For the following week, they have been conducting the next experiment and gathering the 
data. At the end of the study plan, the analysis of the data obtained was made with the FACETS 3.71.4 program, 
developed by Linacre (1993) and based on the Rasch measurement model. 


Research Results 


The three facets are laboratory experiments, criteria, and student groups as a jury. Table 2 represents the 
related data as the calibration map. 
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Measr +Laboratory experiments +Criteria +Group RATING 
2 + + (5) 
E8 
E7 
1 + + 
E3 C13. C15 
E5 - 
C14 
C10 C6 4 
E6 G4 G5 
C20 C5 G8 G9 - 
C17 C4 
FQ * C1 C2 G2 G3 G6 3 
E1 C23. C24 
C16 C21 - 
E4 C11 C22 
C12 C7 G1 
C18 2 
C19 G7 
E2 
1+ +) 
Measr +Laboratory experiments +Criteria +Group RATING 


The science teacher candidates scored five laboratory experiments above the intermediate level, according to 
the “measr” on the left side of Table 2. Table 3 provides more information on related logit values in greater depth. 


Table 3 


Logit Values for Three Facets: Laboratory Experiments, Criteria and Group 


Laboratory experiments Criteria Logit Group Logit 
E8 C13 93 G5 33 
E7 C15 88 G4 27 
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Laboratory experiments Logit Criteria Logit Group Logit 
E3 90 C14 50 G8 25 
E5 80 C6 40 G9 23 
E6 25 C10 36 G6 .00 
E1 -.07 C5 21 G3 -.04 
E4 -.34 C20 15 G2 -.05 
E2 -.10 C4 ah G1 -.38 

C17 06 G7 -.61 
C3 05 
C2 -.01 
C1 -.02 
C24 -.06 
C23 -.10 
cg -.21 
C16 -.21 
C21 -.21 
C22 -.27 
C11 -.29 
C7 -44 
C8 -44 
C12 -.44 
C18 -.46 
C19 -.56 


According to Table 3, there is greater success with the science activity labeled as E8 (logit value: 1.43). Less 
successful is the science activity with code E2 (logit value: -.70). C19 (logit value: -.56) is the hardest criterion, 
whereas C13 (logit value:.93) is the easiest. With a logit value of.33, the G5 jury is the most forgiving, and the G7 
panel (logit value of -.61) is the harshest. 


Laboratory Experiments Performance Analysis 


Table 4 displays information regarding the performance analysis of the lab experiments, including the ob- 
served average, total score, and logit value. 


Table 4 
Laboratory Experiments Performance Analysis 


phe Number Measure pe Infit ZStd Outfit Std Bi pies 

E8 8 1.43 4 82 7 83 6 1019 472 
E7 7 1.32 40 1.02 4 1.65 24 1009 4.67 
E3 3 90 07 92 5 65 21 946 4.38 
ES 5 80 7 86 AA 88 6 925 4.28 
E6 6 25 05 1.12 13 1M 9 757 3,50 
Et 1 -07 05 9 3 87 13 629 2.91 
EA 4 31 05 89 AA 93 = 534 2.47 
E2 2 70 06 1.22 18 1.13 7 406 1.88 
RMSE (Model) =.07 2 =7998 df=7 —_p<.001 Reliability =.99 
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Table 4 shows that the dependability coefficient is.99 and the RMSE (Model) is.07. The science activities differ 
from one another in a quantifiable way. This hypothesis has a separation index of.07, placing it under the fixed ef- 
fect category. The chi-square test was used to evaluate the reliability coefficient of .99 (y?= 799.8, df = 7, p <.001). 
Rejecting the null hypothesis was done. It indicates that the actions differ significantly from one another statisti- 
cally. The tasks in the qualification sequence are completed in the following order: E8, E7, E3, E5, E6, E1, E4, E2. 


Criteria Analysis 


The reliability coefficient is .92, and the separation index is 3.46. The standards by which science-related 
activities are judged diverge significantly. Chi-square was used to test this hypothesis (y* = 246.0, df = 23, p <.001). 
Rejecting the null hypothesis was done. These findings indicate that the criteria used to rate science activities dif- 
fer statistically significantly. Table 5 displays specifics on the criteria measurement analysis, such as the logit value, 
total score, and observed average. 


Table 5 
Criteria Measurement Report 


Criteria Meas. S.E Infit ZStd Outfit ZStd Total score Obs. Aver. 
C13 93 15 96 0 1.32 8 327 4.54 
C15 88 14 1.18 iE 1.27 Jl 325 4.51 
C14 50 2 1.40 1.8 2.22 2.7 302 4.19 
C6 40 2 1.35 1.7 1.72 18 295 4.10 
C10 36 M1 16 -1.3 86 -3 292 4.06 
C5 21 AM 1.28 1.4 1.57 1.6 280 3.89 
C20 AS AM 95 -.2 86 -.3 275 3.82 
C4 Al M 16 -1.4 68 “1.4 271 3.76 
C17 06 M 1.04 2 19 -6 267 3.71 
C3 05 MM 91 -4 1.23 8 266 3.69 
C2 -.01 MM 95 -.2 95 0 261 3.63 
C1 -.02 MM 82 -.9 91 -.2 260 3.61 
C24 -.06 10 1.28 1.5 1.11 A 256 3.56 
C23 -.10 10 1.04 2 19 -7 253 3.51 
C9 -.21 10 1.08 5 94 -.1 242 3.36 
C16 -.21 10 91 -4 14 -9 242 3.36 
C21 -.21 10 1.09 5 19 -7 242 3.36 
C22 -.27 10 82 -9 66 -1.3 237 3.29 
C11 29 10 81 1.0 67 -1.3 235 3.26 
C7 -44 10 1.00 0 89 -.3 224 3.11 
C8 -44 10 1.05 3 95 -1 224 3.11 
C12 -.44 10 84 -8 85 -5 221 3.07 
C18 -.46 10 88 -6 65 -1.4 219 3.04 
C19 -.56 10 91 -4 16 -.8 209 2.90 


RMSE (Model) =.11 ¥2=246.0 df=23 p<.001 Reliability =.92 


Table 5 shows that the simplest criterion is convenient materials. The criteria are listed from the simplest 
to the hardest: convenient materials: C13, reliability tested (at least 3 times): C15, step by step movement: C14, 
well-explained hypothesis: C6, design is for problem solving: C10, hypothesis and problem are parallel: C5, correct 
measurement calculation: C20, solvable problem: C4, convenient tables/graphics: C17, well-content problem: C3, 
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well-explained problem: C2, qualitative problem: C1, hypothesis based evaluation: C24, well-explained content 
results: C23, correct controlled variable: C9, correct variables are controlled: C16, correct tables with variables: C21, 
results explain the problem: C22, changing independent variable: C11, correct dependent variable: C7, correct 
independent variable: C8, measuring dependent variable: C12, tabled variables: C18, correct units of variables: C19. 


Jury Analysis 


Table 6 provides information regarding the jury’s analysis, including the logit value, total score, and observed 
average. The jury was composed of science teacher candidates. 


Table 6 
Group Measurement Report 


Group Nu Measure Exp. S.E Infit ZStd Outfit ZStd Total Score ree 
G5 5 33 07 95 -4 al -1.5 766 3.99 
G4 4 27 07 96 -3 1.10 0 755 3.93 
G8 8 25 07 JT -2.1 69 -1.7 749 3.90 
G9 9 23 07 95 -4 88 -.6 745 3.88 
G6 6 .00 07 1.03 3 1.58 2.8 694 3.61 
G3 3 -.04 07 87 -1.1 86 -1 685 3.57 
G2 2 -.05 07 1.31 25 1.08 A 683 3.56 
G1 i -.38 06 1.39 3.1 1.15 9 602 3.14 
G7 7 -.61 06 68 -3.2 1.02 Al 546 2.84 


RMSE (Model) =.07 32=191.1 df=8 p<.001 Reliability =.95 


The result for the reliability coefficient is.95. 4.69 is the group jury separation index. When the hypothesis 
“there is a difference between severity/leniency of the group jury” was tested using a chi-square test (y?=191.1, 
df=8, p<.001), the null hypothesis was rejected. Table 6 shows that the group jury with code G5 is the most lenient, 
while the group jury with code G7 is the severest. Judges are ranked in G7, G1, G2, G3, G6, G9, G8, G4, and G5 in 
order of leniency to severity. 


Jury Bias Analysis 


Table 7 presents the group juries, logit values, observed scores, and anticipated scores as part of the bias/ 
interaction report. 


Table 7 
Bias/Interaction Report 


Obs. Score Exp. Score Obs. Count Obs-Exp Average Group measr SciAct. Proj Measr+ 
75 105.07 24 -1.25 G7 -.61 E8& 1.43 
40 82.69 24 -1.78 G2 -.05 E6 25 
64 95.93 24 -1.33 G1 -.38 E3 90 
98 113.08 24 -.63 G6 .00 E7 1.32 
71 96.21 24 -1.05 G4 27 E6 25 
61 84.97 24 -1.00 G5 33 E1 -.07 
97 109.53 24 -.52 G8 25 E5 80 
38 56.29 24 -.16 G3 -.04 E4 -.31 
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Obs. Score Exp. Score Obs. Count Obs-Exp Average Group measr SciAct. Proj Measr+ 
91 103.59 24 52 G3 -.04 E5 80 
107 114.03 24 -.29 G6 .00 E8 1.43 
40 55.38 24 -.64 G5 33 E2 -.10 
119 110.99 24 33 G8 25 E3 90 
119 109.51 24 40 G1 -.38 E8& 1.43 
111 98.05 24 54 G5 33 E6 25 
109 95.17 24 58 G8 25 E6 25 
44 34.41 24 40 G1 -.38 E2 -.10 
101 82.54 24 AT G4 27 E1 -.07 
104 84.88 24 80 G6 00 E6 25 
57 42.15 24 62 G2 -.05 E2 -.10 
70 52.12 24 15 G1 -.38 E1 -.07 
75 55.93 24 19 G2 -.05 E4 -.31 
110 83.09 24 1.12 G3 -.04 E6 25 
120 105.97 24 58 G3 -.04 E3 90 
120 111.37 24 36 G4 27 E3 90 
120 112.02 24 33 G5 33 E3 90 
120 110.75 24 39 G5 33 E5 80 
120 115.58 24 18 G5 33 E7 1.32 
120 102.38 24 oy 6) G7 -.61 E7 1.32 
120 113.62 24 27 G2 -.05 E8 1.43 
120 113.70 24 26 G3 -.04 E8 1.43 
120 115.87 24 A7 G4 27 E8 1.43 
120 115.71 24 18 G8 25 E8 1.43 
120 115.61 24 18 G9 23 E8 1.43 

86.5 86.47 24.0 .00 Mean (Count:72) 
28.6 26.15 0 52 S. D. (Population) 
28.8 26.33 0 52 S. D. (Sample) 


Fixed (all = 0) 7 = 292.8 df=72 significance (probability) < .001 


Table 7 indicates that some groups could be very strict or very lenient when it comes to science-related ac- 
tivities. For instance, in the science activity, G3 (coded as E4) received 38 points, although the predicted score was 
56.29; in the same way, G2 (coded as E4) received 75 points, whereas the expected score was 55,93 points. The 
expected score for the science activity was 105.07 points, but G7 (classified as E8) only received 75 points. Figure 
1 and Table 8 indicate the qualitative results. 
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Figure 1 


Pre-Service Science Teachers’ Views About Peer/Self-Assessment 
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Qualitative Results on Science Teacher Candidates’ Views about Peer/Self-Assessment 


Categories and Codes 
Evaluation 


¢ Objective 


¢ Reliable 


© Subjective 


Application 


© Positive 


¢ Negative 


e Biased 


Quotes 


P6: Yes, we can describe ourselves as a reliable and objective evaluator. When evaluating our own performance 
objectively, we try to be honest and observe every aspect from a critical perspective. Likewise, we try to evaluate 
the performance of our colleagues or other teachers objectively. 

P5: Of course, yes. Since we know what the report our teacher wants is like, we pay attention to it. We evaluate 
our friends’ opinions just as objectively. We never treat anyone biased. 


P8: While doing peer self-evaluation, we did not consider our sincerity with anyone and evaluated only based on 
the experiment report. That's why we think we are objective and reliable. 
P1: When making an evaluation, | only look at the evaluation sheet, not the person doing it. 


P2: We can't say it's very good. Because sometimes we can make comparisons while evaluating, or some criteria 
may seem inadequate or inadequate to us. We cannot say that it is objective and reliable for sure. 

P4: We evaluated the report by preparing an experiment in which we converted motion energy into light energy. 
Some groups were not objective. 


P7: Ona positive note, we saw our shortcomings through the eyes of our peers because of multiple votes. 

P3: Our positive aspects are that we maintain our respect for each other. We respect each other and respect your 
opinions. We distribute the work equally to everyone. There are no under- or over-employments among us. Our 
negative aspects are almost non-existent when we look at them in general. If we have only one flaw, it would be 
that we do it slowly. 


P1: | don't think everyone scores fairly. 

P6: It can sometimes be difficult to make personal self-criticism and loss of motivation may occur in the process. 
Additionally, some teachers may feel that the evaluation process is unfair. However, we think that, in general, these 
practices support the professional growth of educators. 


P2: The adequacy of the evaluation criteria may vary from person to person. In other words, while the parts of 
the report are sufficient for me, they may be insufficient for someone else, or vice versa. Sometimes our friends 
can act biased. 
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Categories and Codes Quotes 
Usage 


e Preference in actual P3: It may vary from subject to subject and grade level, but we plan to use it in general. It is useful because it 
introduces the student to his strengths and weaknesses. It also teaches us to take responsibility. 
P7: In my opinion, it should be used because it increases cooperation and interaction in the classroom with peer 
evaluation. It makes students pay more attention. It helps students realize their strengths and weaknesses. 


e Preference intheory  P1: Even if! did peer evaluation, | wouldn't add it to the actual score because | didn't trust the students to keep 
their emotional side in the background. 
P2: We can consider using it because students will have experience, but we may obtain a lower evaluation rate 
because they may think biased. 


Experience 

¢ Developmental P6: This process helped me improve my teaching skills and guide my students more effectively. 
P5: Measurement and evaluation, as the literal meaning of the word, is a subject we have mastered. We also had 
the chance to experience practical lessons. For example, evaluating other groups and us in science teaching or 
laboratory classes. 

¢ Negative P7: We didn't have enough time. We couldn't do a detailed review because it was rushed. We could not understand 
what was expected from us while making the evaluation, so we had difficulty. 

e Min. error P2: We tried to evaluate the items as much as we could, but some items were left between two points, and we 
gave points to some of them by comparing them with other groups. 

Discussion 


In the current study, the multi-facet Rasch measurement model was used in the quantitative dimension 
to examine the opinions of science teacher candidates regarding their ability to prepare experiments based 
on science process skills, and the MAXQDA-20 software was used in the qualitative dimension to do the same. 
The simultaneous surfaces employed in the research (experiments based on strictness/generosity of the juries, 
adequacy of the materials used, and science process skills) were ranked among themselves as a result of the 
Rasch measurement model. Accordingly, out of eight laboratory experiments based on science process skills, 
experiment 8 had the highest quality and experiment 2 had the lowest quality. On the other hand, when the 
items prepared regarding the experiment preparation criteria depend on science process skills were examined, 
“The materials in the experiment are suitable for the problem and design of the experiment.’ (C13) and “At least 
three tests were made to ensure the reliability of the experimental data.” (C15) were the easiest items. The most 
difficult criterion to fulfill was “The units of the dependent and independent variables are written correctly.’ 
(C19). When research in the literature was examined, controversial results were encountered in using science 
process skills in science laboratory experiments. While some studies presented science teacher candidates to 
be very successful in “Correct units of variables” (Saka, 2019; Koyunlu Unlu, 2020), some studies presented them 
as unsuccessful (Govindasamy, Samsudin & Bakar, 2015; Muslu Kaygisiz, Zirve & Ucar, 2017). Numerous studies 
have consistently found that students exhibit a deficient proficiency in science process skills (e.g., Aydogdu, 
2015; Irwanto et al., 2017; Ozgelen, 2012). Tilakaratne and Ekanayake (2017) found in another study that pupils’ 
basic process skills were poor. Furthermore, Oztiirk, Tezel, and Acat (2010) noted that pupils’ control over vari- 
ables and inferential abilities were lacking. Supporting this research finding, Irwanto and Prodjosantoso (2018) 
posited that science process skills could be developed to determine the dependent and independent variables 
used in an experiment. When the jury’s strictness/generosity knowledge regarding designing laboratory experi- 
ments based on science process skills was evaluated, G5-coded jury members displayed the “most generous” 
behaviour, and G7-coded jury members exhibited the “strictest” behaviour. Supporting this conclusion, various 
rater characteristics produced statistically significant differences between raters in numerous studies in the 
literature (BastUrk, 2008; BastUrk, 2010; Kése et al., 2016; Mumpuni et al., 2022; Semerci, 2012; Semerci, 2011; 
Semerci et al., 2013; Uyanik et al., 2019; YUziak et al., 2015). 

Within the scope of the qualitative results of the research, participants’ positive/negative opinions about 
application, experiences, reliability/objectivity, and future usage tendencies regarding the peer and self- 
evaluation process were examined. Teacher candidates saw themselves as reliable and objective evaluators. 
When evaluating themselves and their peers, they made evaluations according to the evaluation sheet, not 
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bilateral relations. Although they made performance according to predetermined criteria for objective and 
reliable evaluations, very few juries added their own opinions to the evaluation process and acted subjec- 
tively. To support this conclusion, the majority of research studies employing the many-facet Rasch model in 
the literature (Akin & Bastiirk, 2012; Atilgan, 2005; Bastiirk, 2008, Bastiirk, 2010; Semerci, 2012, Semerci, 2011; 
Semerci et al. al. 2013; Uyanik et al. 2015) show that raters are not always neutral. Half of the participants stated 
that there was no negative aspect of the process and that their shortcomings were evaluated and expressed by 
their friends in the process. However, the other half of the participants stated that they had problems making 
self-criticism and maintaining their motivation. In addition, the criteria were sometimes inadequate, and the 
evaluators were biased. Therefore, by going over and editing these items, the assessment forms can be made 
even better. According to the findings, analysing unexpected reactions might greatly enhance peer and self- 
evaluation procedures. Most participants stated that using it in their future classes would increase interaction 
and cooperation among students. It also offers students the chance to improve themselves by showing their 
weaknesses and strengths. Although participants think that using it theoretically will be effective in increas- 
ing students’ experience, they believe that using peer evaluation as a score will cause prejudice because it 
has an emotional dimension. Results from previous studies in the literature were discovered to corroborate 
the conclusions of this study. Teacher candidates saw themselves as reliable raters when performing peer and 
self-evaluation, and they also gained experience and had positive opinions about the process (Demir, 2023). 
Self-assessment and peer assessment have been found to enable participants to collaborate and communicate 
effectively and support content knowledge development (Fang et al., 2021; Sahin-Taskin, 2018; Tait-McCutcheon 
& Knewstubb, 2018). 

In terms of experience, participants stated that they improved their teaching skills, that they would be 
more helpful to their students, and that they felt more expert in assessment and evaluation from a develop- 
mental perspective. The negative experience was that the process was fast and when there was a difference 
between two points in the evaluation, they tried to make minimum mistakes by comparing the performances 
of the groups and giving points. Raters may agree on aspects unrelated to the actual measurement. Therefore, 
it was essential to gain a deeper understanding of the rating process, including how raters approach the task, 
the factors they consider, and the characteristics that influence their rating behaviour. Lumley’s (2005) model 
illustrates that raters played a crucial role in the evaluation process. This process was marked by conflict and 
challenge, as raters brought their unique personalities and past experiences to the rating task. Understand- 
ing how these individual traits and histories shape their evaluations was vital to comprehend the impact they 
have on the outcomes of their ratings. As science teachers tasked with assessing students’ science lab experi- 
ment performance, they would inevitably judge the students’ work, whether it’s an experiment, a portfolio, or 
a written assignment. It is crucial to recognize that the act of judging can introduce undesirable variations or 
mistakes in the assessment process, which can impact the quality of the students’ evaluations (Govindasamy 
et al., 2015). Therefore, it was important to scrutinize and be aware of the judging behaviour to ensure fair and 
accurate ratings of students’ performances. 


Conclusions and Implications 


The results of this research may offer encouragement and promise. The study was conducted using both 
quantitative and qualitative methods to provide compelling and well-founded explanations for the obtained 
findings. However, a limitation of this research is the small number of teacher assessors. To enhance the validity, 
studies should be conducted across various schools and locations within the country. Additionally, considering 
different school categories (such as vernacular, national, and boarding schools) and the location types (urban, 
semi-urban, and rural) can significantly impact the findings. Another limitation is the limited number of items 
(24 items) evaluated by the raters focused on a single topic. Future research should expand the number of items 
and encompass a broader range of topics within the laboratory syllabus for science process skills in science 
education. Assessments should be conducted over a longer duration to substantiate the findings, ensuring 
that each examinee has ample opportunity to demonstrate their capabilities. Lastly, the small sample size 
precludes definitive conclusions about the reliability of rater assessments. Subsequent studies should aim to 
address this critical issue. Rasch analysis produced a reliability coefficient that resembled both the KR 20 and 
Cronbach alpha reliability values. When the analysis results and reliability coefficients were evaluated together, 
the groups scored consistently, and the form prepared within this framework served the purpose. 

As a result of all of these, the Multi-Facet Rasch measurement model could be used effectively to evaluate 
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peer groups in science education and objective results could be obtained. At the end of the related research, it 
can be easily seen that Multi-Facet Rasch Analysis gives the reliable results and provides objectivity throughout 
the evaluation process. With the help of that, science teacher candidates have been also able to see their level 
based on the criteria and peer assessments. It can be concluded that in terms of teaching laboratory activities 
that are based on the science process skills, while some of the science teacher candidates have the necessary 
qualifications, some of them don't have it. Therefore, it has been observed that the science teacher candidates 
who have less successful results should be focused more on the following educational processes. 
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Abstract. An important educational goal 
is enabling students to learn scientific 
concepts. The scientific concepts learned 

in class are developed within students’ 
cognitive structures. Despite the successful 
application of Semantic Network Analysis 
(SNA) to study these cognitive structures, 
there has been limited examination of 
students’ concept networks based on their 
individual characteristics. Therefore, this 
study aims to evaluate the differences in 
students’ characteristics based on their 
cognitive levels, which influence their 
thinking and behavior. To analyze these 
differences, this study compares concept 
and connected concept networks, focusing 
on photosynthesis, a challenging life 
sciences topic. The study’s results indicate 
that students could not clearly distinguish 
concepts by subtopic, but there were 
changes in the concept network after class. 
Although the types and number of concepts 
students knew were similar depending on 
their cognitive level, the concept network 
structure differed. Additionally, some 
students could not distinguish between 
similar concepts. Thus, teachers are advised 
to differentiate between similar concepts 
during instruction and address personal 
variables such as students’ cognitive levels. 
Keywords: cognitive level, concept 
network, connected concept network, 


learning concept, photosynthesis 
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Introduction 


Science education can be said to be a meaningful interaction between 
students, science teachers, and scientific content. Among these, it is very dif- 
ficult to control student-related variables. Therefore, it is very important to un- 
derstand how students perceive scientific content, think, and construct new 
concepts. When learning science content, most students learn with scientific 
concepts. Accordingly, the purpose of science education is to provide learners 
with an understanding of related concepts (You et al., 2002). Therefore, the 
science curriculum in the Republic of Korea aims to help students understand 
these concepts and develop their capabilities (MOE, 2022). The nature of 
scientific concepts makes it difficult to acquire knowledge independently; 
thus, students learn most concepts in class through teachers (Mallow, 1986; 
Pianta & Hamre, 2009; Schenke et al., 2017). 

Concepts taught in class are delivered and absorbed by learners using 
their cognitive structure. At this time, most teachers have focused primarily 
on conveying conceptual knowledge without considering how learners un- 
derstand and construct concepts in class (Widodo & Duit, 2002; Widodo et al., 
2002). However, it is known that differences between the content taught by 
teachers and the level of understanding of students can lead to misconcep- 
tions and errors (Resnick & Klopfer, 1989). Therefore, in order for teachers to 
effectively present concepts, there was a need to analyze and present which 
concept structure (i.e., connection structure between concepts) is optimal 
for learning. In addition, because the learner’s characteristics have a very 
great influence on learning, knowledge of how to transform and reorganize 
the content to suit the learners’ characteristics and introduce it into classes 
has also been very important (Schulman, 2013). Therefore, it is necessary to 
classify and identify learning concepts according to the characteristics of 
students and to find ways to teach them. 

Semantic Network Analysis (SNA) is widely used to analyze how learners 
recognize and construct concepts within their cognitive structures. It has been 
used to analyze the structure of concepts presented in science textbooks 
(Chung et al., 2018; Kim & Kwon, 2016) and the connection between curricula, 
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textbooks, classes, and assessments (Kim et al., 2019). Some studies have analyzed concept structures according 
to students’ listening types (Lim et al., 2020). In addition, network analysis has been used to study the connection 
between the concepts taught by teachers and those learned by students (Kim et al., 2023). However, these studies 
have failed to consider the learners’ characteristics. Therefore, this study attempted to use the learners’ cognitive 
level as a variable, which is an important factor in examining the learners’ characteristics in science education. 

Thinking and behavior are determined by cognitive level; Therefore, it is known that if a teacher fails to com- 
municate scientific content in a way that matches students’ cognitive levels, their teaching will be ineffective (Kim 
et al., 2024; Kwon et al., 2012). Although students’ cognitive levels are crucial for planning and delivering teach- 
ing, there is a lack of research on the concepts students understand based on their cognitive level and how these 
concepts are structured. Therefore, this study analyzes the concepts students have learned and the relationships 
between them based on cognitive levels. 

The knowledge gained through education is expected to be applied, continuously used, and maintained 
(Baldwin & Ford, 1988). In simpler terms, students are expected to actually use what they have learned, create 
change, and consistently display changed behavior or applied knowledge. Accordingly, it has been important 
to assess whether the knowledge acquired in class can be applied and maintained in other fields (Chance, 2009). 
Therefore, this study analyzes whether learned concepts were applied and maintained by investigating if they were 
learned immediately after class and following up on whether this knowledge was maintained four weeks later. 

The topic of photosynthesis is used to compare students’ concept networks based on their cognitive level. 
Photosynthesis is known as one of the most essential concepts because it is a fundamental concept for under- 
standing metabolism, a basic biological phenomenon (Kijkuakul et al., 2006). It is a crucial topic in biology and is 
widely studied in elementary and lower and upper secondary schools in the Republic of Korea. It is also known 
to be a difficult topic for both learners and teachers (Ross et al., 2005) because it comprises complex concepts 
(Marmaroti & Galanpoulou, 2006). Therefore, it is an appropriate topic to analyze the connections and differences 
between concepts recognized by students after class through concept network analysis based on cognitive level. 

Therefore, the purpose of this study was to obtain implications on how to present concepts in order to teach 
science topics that students find difficult depending on their characteristics. Accordingly, this study attempted to 
understand the structure of the concepts that students learned through classes on the topic of photosynthesis 
being covered in lower secondary schools of the Republic of Korea and suggest teaching - learning strategies 
utilizing this. The research questions of this study are as follows. 

Depending on the variable of students’ cognitive level, what is the difference in the network of students’ 
learning concept immediately after conducting a class on the topic of photosynthesis and after 4 weeks of class? 
Photosynthesis is divided into three subtopics: reactants, products, and environmental factors. The concept structure 
learned by students is specifically developed through learning concept networks and connected concept networks. 
Through this, specific teaching - learning strategies and related implications are expected to be derived, helping 
teach students photosynthesis based on their cognitive levels. 


Theoretical Background 
Photosynthesis-related Content presented in the Curriculum 


Currently, lower secondary schools in the Republic of Korea use the national curriculum, which was revised 
in 2015. The revised science curriculum teaches photosynthesis in the 6th, 8th, and 11th grades. The specific units 
presented in each grade are as follows (MOE, 2015):’Structure and Function of Plants’ (6 grade),’Plants and Energy’ 
(8 grade), and ‘Cellular Respiration and Photosynthesis’ (11 grade). The ‘Structure and Function of Plants’ unit 
helps students understand the individual functions of plant organs (e.g., roots, stems, leaves, and flowers) and that 
they are interconnected. The inquiry activity focuses on identifying photosynthetic products, and the key concepts 
are “transpiration” and “photosynthesis.” 

The ‘Plants and Energy’ unit focuses on understanding how plants create and use nutrients to obtain the 
necessary energy. The learning content related to photosynthesis includes ‘the role of photosynthesis in produc- 
ing nutrients;‘substances necessary for photosynthesis; and ‘products: The research activity explores the environ- 
mental factors that affect photosynthesis, and the key concepts are “the materials necessary for photosynthesis,” 
“photosynthetic products,"“factors affecting photosynthesis,” and “production, storage, and use of photosynthetic 
products.’ Based on this, this study conducts research through classes focusing on three subtopics:‘photosynthesis 
reactants, ‘photosynthesis products, and ‘environmental factors. 


659 


https://doi.org/10.33225/jbse/24.23.658 


Journal of Baltic Science Education, Vol. 23, No. 4, 2024 


LEARNING GCONCEPT NETWORKS IN THE PHOTOSYNTHESIS BASED ON STUDENTS’ ISSN 1648-3898 = /erint, 


COGNITIVE LEVELS ISSN 2538-71 38 /Online/ 


(PP. 658-678) 


The ‘Cellular Respiration and Photosynthesis’ unit helps students understand the metabolic relationship be- 
tween cellular respiration and photosynthesis. Inquiry activities focus on researching the history of science related 
to photosynthesis and separating leaf pigments. The key concepts include “the structure and functions of chloro- 


mn. mn, 


plast,"“carbon fixation in photosynthesis,"“the electron transport chain,’and “light reaction through photosystems.” 
Students’ Cognitive Levels in the Republic of Korea 


Logical thinking was first introduced as a research topic in the Republic of Korea in 1978. It was analyzed 
using a questionnaire developed by Roadrangka et al. (1983), which Choi et al. (1985) translated into Korean. The 
determined cognitive levels of lower secondary school students in the Republic of Korea differed from study to 
study. Choi and Hur (1987) reported that 68% fell into the concrete operational stage of cognitive development, 
and only 4.4% were in the formal operation stage. Kim et al. (2002) reported that the concrete operational stage 
comprised 37.8%, the transitional stage encompassed 32.7%, and the formal operational stage accounted for 
29.5%. Thus, combining previous studies, 23.0-63.1% of lower secondary school students in the Republic of Korea 
fall in the concrete operational stage, and 7.4-36% are in the formal operational stage. 

Existing research in the Republic of Korea has been conducted on the differences and effects of scientific 
achievement and inquiry abilities depending on cognitive level, as well as on the relationship between cognitive 
levels and textbook content (Choi et al., 1985; Jeon & Park, 2014; Kim & Kim, 2009; Kim & Lee, 2001). Most of these 
studies showed that formal operational students have high science achievement and inquiry skills, and their teachers 
are effective. However, many studies have reported that Korea’s science curriculum content was beyond students’ 
cognitive levels (Kang et al., 2012; Kim et al., 2004; Jeong & Jang, 2017; Yang et al., 2024). 


Research Methodology 
Design 


This study was conducted using a quantitative research design. The quantitative research design involves 
collecting quantitative statistical data on several variables studied (Cohen & Manion, 1994; Creswell, 2003). This 
study is a quantitative research design that collects data using questionnaires. 

The independent variable in this study was students’ cognitive levels, and the dependent variables were 
learning concept networks in the photosynthesis. Quantitative data of this study was collected by modifying and 
supplementing the questionnaire developed by Jeong and Kim (2011) and Lim et al. (2020) to suit the photosyn- 
thesis context. The study was conducted from May to July 2023. The participants were students from three lower 
secondary schools in a D metropolitan city and P city with a population of 2.5 million in the Republic of Korea and 
were 8th grade students who were not gifted or special students. 

Under the scope of this study, the participants were invited through the science teacher at the lower second- 
ary school. All of the participants and their parents in this study agreed to participate voluntarily. While collecting 
the data, the participants did not write their names while responding to the items. 


Participants 


The participants were from three lower secondary schools located in a D metropolitan city and P city with 
a population of 2.5 million in the Republic of Korea. The three schools were ranked mid-level as a result of the 
national academic achievement evaluation in the Republic of Korea. The researchers explained the purpose and 
method of the study in advance, which all of the participants and their parents agreed to. Participants comprised 
three science teachers (one male and two female) from three lower secondary schools and their 8""-grade students 
(aged 13 and 14). One teacher had four years of teaching experience, and two had more than ten years; out of the 
three, two had completed master’s degrees. 

Students participating in the study were asked to fill out a questionnaire on either ‘photosynthesis reactants, 
‘photosynthesis products, or ‘environmental factors’ to reduce problems caused by conducting the questionnaire 
multiple times. As a result, 104 participants (male: 50, female: 54) filled out questionnaires on ‘photosynthesis 
reactants, 102 (male: 45, female: 57) filled out ‘photosynthesis products, and 86 (male: 41, female: 45) filled out 
‘environmental factors: Among these, students who left responses blank or responded indifferently to any of the 
cognitive level or photosynthesis learning concept questionnaires were excluded from the analysis. As a result, 98 
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(male: 45, female: 53) students were analyzed for ‘photosynthesis reactants, 85 (male: 36, female: 49) for‘photosyn- 
thesis products, and 68 (male: 28, female: 40) for ‘environmental factors. Likewise, the gender ratio was not biased 
towards one side. Table 1 is gender composition based on participants’ cognitive level. 


Table 1 
Gender Composition based on Participants’ Cognitive Level 


Concrete Operational Stage Transition Stage Formal Operational Stage 
gg Total 
Male Female Male Female Male Female 
Photosynthesis Reac- 97 97 13 17 5 9 98 
tants 
Photosynthesis Products 20 28 11 15 5 6 85 
Environmental Factors 14 24 11 11 3 5 68 


Regarding students’ cognitive levels, 54 (55.1%) were in the concrete operational group and 14 (14.3%) in 
the formal operational group for the ‘photosynthesis reactants’ questionnaire. For the ‘photosynthesis products’ 
questionnaire, 48 (56.5%) were in the concrete operational group, and 11 (13.0%) were in the formal operational 
group. For the ‘environmental factors’ questionnaire, 33 (55.9%) were in the concrete operational group, and 8 
(11.8%) were in the formal operational group. In this study, students in the concrete operational stage account for 
55.1-56.5%, and students in the formal operational stage comprise 11.8-14.3%, which is within the composition 
range of previous studies in the Republic of Korea. 

Research conducted in connection with normal educational practice in schools and educational institutions de- 
termined andannounced by the Minister of Health and Welfare with official permission from the head of the educational 
institution in accordance with Article 2 of the Elementary and Secondary Education Act and Article 2 of the Higher Ed- 
ucation Act is research excluded from research on human subjects under the Bioethics Act (KONIBP, 2018). In addition, 
the data collected to classify students’ cognitive levels was used only for research purposes, and consent was obtained 
after notifying the school, students, and parents that students’ personal information would not be collected or recorded. 


Class Contents of Photosynthesis 


Classes of photosynthesis were taught by the science teachers at each school according to the class plan of 
the respective schools. The classes took place from May to July 2023. The concept network for photosynthesis was 
classified into three subtopics for analysis: ‘photosynthesis reactants,’photosynthesis products, and ‘environmental 
factors. Each subtopic was covered in a separate session; thus, three sessions of classes with teaching concepts, 
including inquiry activities, were conducted. 

The class of the ‘photosynthesis reactants’ covered the photosynthesis process, in which plants use light energy 
to create nutrients from carbon dioxide and water. The reason why photosynthesis is necessary was presented 
by comparing and explaining the differences in how animals and plants obtain the energy they need to live. By 
explaining the experiment on determining the substances necessary for photosynthesis by observing the color 
change of the BTB solution, the process of the experiment can be understood in connection with photosynthesis. 

The class of the ‘photosynthesis products’ studied plants’ glucose and oxygen production using light energy, 
carbon dioxide, and water. In class, the teacher explained an experiment where the starch produced through the 
photosynthesis process in Hydrilla verticillata leaves is observed, suggesting that photosynthesis occurs when 
glucose is converted into starch and temporarily stored in chloroplasts. The teachers also explained that oxygen is 
generated from photosynthesis using an experiment to collect and identify gas generated during photosynthesis. 

The class of the ‘environmental factors’ examined the overall process of photosynthesis from previous classes 
to determine how the amount of photosynthesis is affected by environmental factors such as light intensity, car- 
bon dioxide concentration, and temperature. The teacher explained an experiment measuring the time it takes 
for spinach leaves to rise depending on the number of lights turned on. Through this, the teacher helped students 
understand changes in photosynthesis based on light intensity. In addition, students were encouraged to think of 
experimental methods to determine changes in photosynthesis according to temperature changes and carbon 
dioxide concentration. 
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Questionnaire 


The photosynthesis learning concept questionnaire used in this study divided photosynthesis into three 
subtopics: ‘photosynthesis reactants, ‘photosynthesis products, and ‘environmental factors: These subtopics were 
modified from and supplemented by inspection tools used by Jeong and Kim (2011) and Lim et al. (2020) to suit 
the photosynthesis context. The questionnaire included subtopic names, learning objectives, related pictures, 
and simple examples of how to complete the questionnaire. After a class on each subtopic, students were asked 
to describe the related content they had learned. The questionnaires were open-ended so students could write 
freely, and an online format was used. 

The students’ cognitive level questionnaire used in this study was group assessment of logical thinking (GALT) 
test that has been widely used to measure logical thinking levels in the field of education. The Cronbach's alpha 
reliability coefficient of the test was recorded at .52, which is considered moderate for use in the study (Lay, 2009). 
The GALT test used in this study consists of twelve items in six subscales measuring logical operations, as noted 
by Roadrangka et al. (1983). The subscale covers conservation reasoning, proportional reasoning, controlling 
variables, probabilistic reasoning, correlational reasoning, and combinatorial reasoning. It comprises 12 items, 
two for each subscale. 


Data Collection and Analysis 


The cognitive level questionnaire was administered to students before class. After the teacher taught students 
about the photosynthesis, the students were administered the photosynthesis learning concept questionnaire. 
This questionnaire was administered to students again four weeks later. The cognitive level questionnaire was 
administered under the supervision of the homeroom teacher, and the testing took up to 25 minutes. 

The photosynthesis learning concept questionnaire was administered online using tablets provided by each 
school under the supervision of the instructor who taught the photosynthesis class. It took students approximately 
10 to 15 minutes to fill it out. 

The cognitive level questionnaire was structured as follows. For questions one to ten, respondents had to 
provide both the correct answer and the reasoning behind it. For questions eleven and twelve, only those that listed 
all possible cases were treated as correct. Regarding scoring, the cognitive level was divided as follows: four or fewer 
items classified the respondent into the concrete operational stage, five to seven items classified the respondent 
into the transitional stage, and eight to twelve items classified the respondent into the formal operational stage. 

The photosynthesis learning concept questionnaires were classified by subtopic and transcribed sentence 
by sentence. The transcribed content was saved as a text (.txt) file. To compare and analyze the concept structure 
according to students’ cognitive levels, the levels were divided into the concrete operational stage, the transitional 
stage, and the formal operational stage, and the sentences presented in the photosynthesis learning concept 
questionnaire were saved as separate text files (.txt). 

The transcribed files were pre-processed using the NetMiner 4.0 program to extract only noun-type concepts. 
The concepts not needed for analysis were removed through pre-processing (Oh et al., 2022). Among the extracted 
concepts, those that were not scientific were excluded, and those with similar meanings were collected and unified. 
Through this process, 73 concepts were selected. 

To avoid researcher subjectivity, five biology education experts who are current science teachers in lower 
and upper secondary schools and hold a teacher's license and doctoral degree in biology education were asked 
to select the concept. Based on this, 47 analysis concepts were selected by at least three out of five teachers. 

Based on these selected concepts, the concept structure was visualized using the NetMiner 4.0 program. The 
1-mode network was used as a concept network to analyze the structure of concepts recognized by students, and 
the 2-mode connection concept network analyzed connection concepts. In the case of a 1-mode network, the size 
of the node increases proportionally depending on the frequency, and the thickness of the link increases propor- 
tionally according to the frequency of co-occurrence between concepts. Furthermore, the eigenvector centrality 
index analyzes how important and influential connected concepts are on the network (Bonacich, 2007); thus, it is 
mainly used when selecting key concepts on a concept network. 
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Research Results 
Concept Networks 
Photosynthesis Reactants 


The concept network for 8th-grade students’ understanding of ‘photosynthesis reactants’ after the initial 
class and four weeks later based on cognitive level is shown in Figure 1. The frequency and eigenvector centrality 
index of students’ learning concepts of ‘photosynthesis reactants’ after the initial class and four weeks later based 
on cognitive level is shown in Table 2. 

Students in the concrete operational stage identified 21 concepts after the initial class (Figure 1a) and 20 
concepts four weeks later (Figure 1b). In the network of concrete operational students after the initial class, “Pho- 
tosynthesis” was connected with “plant,““leaf“nutrition,;"“light energy,’“oxygen,"“stoma,’ and “hydrilla verticillata.” 
“Light energy” was connected with “water, “plant,” “photosynthesis,’ “oxygen,” and “stoma.” “Carbon dioxide” was 
connected with “absorption,;"“bromothymol blue/“stoma,’“water,’and “leaf.” In the network of concrete operational 
students four weeks later, non-scientific concepts such as “glucose” were presented with high frequency. The network 


WH 


was structured around “photosynthesis,"“carbon dioxide,’ and “light energy.’“Photosynthesis” was connected with 


mu. mn, 


“chloroplast,“plant,light energy,"“oxygen,’and “starch.’“Carbon dioxide” was connected with “bromothymol blue,” 


wn, wn 


“light energy,’“water,’“glucose,’“starch,’ and “photosynthesis.’“Light energy” was connected with “carbon dioxide 


concentration,’“plant,’“bromothymol blue,’ and “photosynthesis.” 


Table 2 
‘Photosynthesis Reactants’ Concept Frequency and Eigenvector Centrality based on Cognitive Level 


Concrete Operational Stage Transition Stage Formal Operational Stage 
Concept After Class pour Nears After Class POU oe After Class ROUFINGS 
Later Later Later 


Freq. E. Freq. E. Freq. E. Freq. E. Freq. E. Freq. E; 


Photosynthesis 66 .256 47 169 57 232 40 252 28 102 25 426 
Carbon dioxide 64 257 43 648 48 651 31 587 20 631 16 493 
Water 46 628 34 649 29 648 21 567 15 578 10 394 
Light energy 41 302 4 331 31 .270 34 513 21 482 18 534 
Glucose 28 026 22 .059 27 019 12 025 1 .010 7 .006 
Oxygen 19 .074 14 .049 12 029 14 025 7 .039 5 002 


* E: Eigenvector centrality index 


Students in the transition stage identified 24 concepts after their initial class (Figure 1c) and 20 concepts four 
weeks later (Figure 1d). In the network of transitional students after the initial class, “Carbon dioxide” was connected 
with“stoma,"photosynthesis,"“water,“plant,’and “light energy:”Photosynthesis” was connected with “leaf;“glucose,’ 
“conversion, “hydrilla verticillata,’ “shift,’ “light energy,’ “plant,’ “absorption,” and “carbon dioxide.’ “Light energy” 
was connected with “photosynthesis,’“water, “absorption, “temperature,” and “hydrilla verticillata.’“Glucose” was 
connected with “starch,’ “oxygen, “hydrilla verticillata”’ and “photosynthesis.” It could be seen that non-scientific 
concepts such as “glucose” existed at the center of the network. And in the network of transitional students four 
weeks later, non-scientific concepts such as “oxygen” were still presented with a high frequency. “Photosynthesis” 


WH wn 


was connected with “water,’“carbon dioxide,"“oxygen,“hydrilla verticillata,’ and “light energy.’“Light energy” was 


mn, mn mh, mn 


connected with “photosynthesis,"water,”“carbon dioxide,"“glucose,plant,’and “light energy.’“Carbon dioxide” was 
connected with “stoma,;"“photosynthesis,’“light energy,’“water,’ and “temperature.” 

Students in the formal operational stage identified 17 concepts after the initial class (Figure 1e) and 16 concepts 
four weeks later (Figure 1f). In the network of formal operational students after the initial class, scientific concepts 
such as “photosynthesis,"“light energy,’ and “carbon dioxide” were located at the center of the network due to their 
high frequency and eigenvector centrality. “Photosynthesis” was connected with “chloroplast,“hydrilla verticillata,’ 


mh, mu, 


“light energy,’"“carbon dioxide,’and “plant’”Light energy” was connected with “hydrilla verticillate,“photosynthesis,” 
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“water, “carbon dioxide,’and“plant,’ Carbon dioxide was connected with “water,“hydrilla verticillata/“light energy, 
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“photosynthesis, and “oxygen.’ In the network of formal operational students four weeks later, scientific concepts 
such as “photosynthesis,’“light energy,’ and “carbon dioxide” were still formed around. “Photosynthesis” was con- 


wu, wu, 


nected with “hydrilla verticillata,’ “storage,’ “chloroplast, 


Wn 


wu 


plant,’“light energy,’ and “nutrition.” “Light energy” was 


mn, 


connected with “carbon dioxide,“water,’“leaf}’ “light intensity,’"“plant,’ and “photosynthesis.’“Carbon dioxide” was 


WH, 


connected with “temperature, 


Figure 1 


water,’ and “light energy.” 
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Photosynthesis Products 


Eighth-grade students’ concept network for ‘photosynthesis products’ after the initial class and four weeks 
later based on cognitive level is shown in Figure 2. And the frequency and eigenvector centrality index of students’ 
learning concepts of ‘photosynthesis products’ after the initial class and four weeks later based on cognitive level 
is shown in Table 3. 

Students in the concrete operational stage identified 21 concepts after the initial class (Figure 2a) and 17 con- 
cepts four weeks later (Figure 2b). The network of concrete operational students after the initial class was structured 


WH, WH, 


around “photosynthesis,” “starch,” “glucose,” and “oxygen”. “Photosynthesis” was connected with “oxygen,’“starch,” 


WH, ww " 


“carbohydrate,” “chloroplast,’“plant,’ and “carbon dioxide.’ “Glucose” was connected with “starch,’ “leaf,” “product, 


wu 


“plant,” and “oxygen.” “Starch” was connected with “carbohydrate, “hydrilla verticillata,’ “iodine potassium iodine 


mu, 


solution;”blue,"“glucose,’and“oxygen.”“Oxygen’ was connected with “starch,""photosynthesis,;""water,’and “carbon 
dioxide”. The network of concrete operational students four weeks later was structured around “photosynthesis,” 
“glucose,’“oxygen,’“starch,’ and “product”. 
Table 3 


‘Photosynthesis Products’ Concept Frequency and Eigenvector Centrality based on Cognitive Level 


Concrete Operational Stage Transition Stage Formal Operational Stage 
Concept After Class Four eens After Class Pour years After Class pour atone 
Later Later Later 


Freq. E. Freq. E. Freq. E. Freq. E. Freq. E. Freq. E. 


Photosynthesis 37 .206 36 .068 42 371 37 140 20 384 16 .265 
Starch 27 ATS 16 323 23 118 20 331 13 A13 7 216 
Glucose 23 490 34 .608 27 310 26 613 10 367 10 AT 
Oxygen 22 491 18 505 15 231 18 514 12 274 "1 442 
Chloroplast 17 .093 10 017 10 161 8 .056 12 325 6 139 
Light energy 16 .098 10 026 24 371 1 036 6 .196 5 .280 


* E: Eigenvector centrality index 
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Students in the transition stage identified 19 concepts after the initial class (Figure 2c) and 21 concepts four 
weeks later (Figure 2d). The network of transitional students after the initial class was structured around “water,” 
“plant,“light energy,’ and “glucose”. “Water” and “light energy,’ which are not scientific concepts, were located at the 


ma mu 


center. “Water” was connected with“plant,"absorption;"glucose,"“light energy,’and “carbon dioxide:“Light energy” 


was connected with “carbon dioxide;"“photosynthesis,’“water,plant,’and “glucose:"“Glucose” was connected with 


“oxygen, “light energy,’ “water, “nutrition,” “product,’ and “starch.” “Starch” was connected with “storage,’ “chloro- 


Ua wn, 


plast,"“iodine potassium iodine solution,’ “product,’ “glucose,” and “nutrition.” The network of transitional students 


wu 


four weeks later, was structured around “photosynthesis,’ “glucose,” and “starch."“Photosynthesis” was connected 


wu, wn, 


with “light energy,’ “chloroplast, “leaf” “glucose, “chlorophyll,” and “plant.’“Glucose” was connected with “starch,” 


mu wn 


“product,’“leaf,’and “photosynthesis.’"“Starch” was connected with “photosynthesis,"“storage,’“glucose,’ and “blue.” 
Students’ formal operational stage identified 19 concepts after class (Figure 2e) and 19 concepts four weeks 
later (Figure 2f). The network of formal operational students after the initial class was structured around “starch,’ 


mn mm 


“photosynthesis,"“glucose,’and “chloroplast. “Photosynthesis” was connected with “chloroplast,plant;“light energy,’ 


wu wu WH wn, 


and “blue.”“Starch” was connected with “chloroplast,” “plant,” “nutrition,” “glucose,’ “oxygen,’ and “carbon dioxide.” 


mm, mm 


“Glucose” was connected with“starch,“nutrition/“product,’and “oxygen.’“Chloroplast” was connected with “plant,” 


ma mu 


“photosynthesis, “iodine potassium iodine solution,"“water,’ and “carbon dioxide.’The network of formal operational 
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students four weeks later was structured around “photosynthesis,” “oxygen,’ and “glucose”. “Photosynthesis” was 
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connected with “plant,’“storage,’“chloroplast,’ and “light energy.’“Oxygen” was connected with “starch,’“product,’” 


mun mw mu, 


“carbon dioxide,"“glucose,’“nutrition,’ and “stoma.’“Glucose” was connected with “oxygen,“nutrition,”“water,’“car- 
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Figure 2 
‘Photosynthesis Products’ Concept Network based on Cognitive Level 
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Environmental Factors 


Eighth-grade students’ concept network for ‘environmental factors’ after the initial class and four weeks later 
based on cognitive level is shown in Figure 3. The frequency and eigenvector centrality index of students’ learning 
concepts of ‘environmental factors’ after the initial class and four weeks later based on cognitive level is shown in 
Table 4. 

Students in the concrete operational stage identified 18 concepts after the initial class (Figure 3a) and 16 
concepts four weeks later (Figure 3b). The network of concrete operational students after the initial class was struc- 
tured around “photosynthesis,“light energy,“light intensity,’and “carbon dioxide’Photosynthesis” was connected 


mn " 


with “oxygen,“plant,“light energy,’"“carbon dioxide,’ and “temperature.”“Light energy” was connected with “water; 
“carbon dioxide;“light intensity,“carbon dioxide concentration,“plant,’and “photosynthesis.’”“Light intensity” was 
connected with “temperature,“light energy,’ and “carbon dioxide concentration.’“Carbon dioxide” was connected 


wa 


with “glucose,"“photosynthesis,"“water,“light energy,’and “temperature.” And the network of concrete operational 
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students four weeks later was structured around “photosynthesis,"“light energy,’“light intensity,’ and “carbon di- 


wn, wu 


oxide.”“Light energy” was connected with “light intensity,’“carbon dioxide concentration,’ “photosynthesis,” and 


mu, mn, mu. 


“carbon dioxide.’“Photosynthesis” was connected with “light energy,’“temperature,;“carbon dioxide;"“oxygen,’and 


“light intensity.’“Light intensity” was connected with “light energy,’“carbon dioxide concentration,’"“temperature,” 
“absorption,’ and “oxygen.” 


Table 4 
‘Environmental Factors’ Concept Frequency and Eigenvector Centrality based on Cognitive Level 


Concrete Operational Stage Transition Stage Formal Operational Stage 
Concept After Class Four Weeks After Class Four Weeks After Class Four Weeks 
Later Later Later 

Freq. E; Freq. E. Freq. E. Freq. E. Freq. E. Freq. E. 
Photosynthesis 38 148 29 158 48 058 46 .133 24 111 17 .088 
Light energy 38 621 36 671 42 688 32 661 17 671 11 654 
Light intensity 22 574 23 658 28 685 28 675 12 .667 12 .690 
Carbon dioxide 21 318 16 165 17 066 10 .037 11 234 6 .019 
Oxygen 17 016 13 063 13 .030 5 .000 6 .008 3 .005 
Temperature 15 114 23 126 28 133 27 .189 10 .008 7 .130 


* E: Eigenvector centrality index 


Students in the transition stage identified 15 concepts after the initial class (Figure 3c) and 18 concepts four 
weeks later (Figure 3d). The network of transitional students after the initial class was structured around “photo- 


wu 


synthesis” and “light energy.’“Photosynthesis” was connected with “chloroplast,“plant,’“light energy,’ and “carbon 


wu Wu 


dioxide.’ “Light energy” was connected with “photosynthesis,’ “plant,” “water,’ and “glucose.” And the network of 
transitional students four weeks later was structured around “photosynthesis,"“light intensity,’ “light energy,’ and 
“temperature.” “Photosynthesis” was connected with “plant,’“temperature,’ “light intensity,’“carbon dioxide,’ and 


“light energy.’“Light energy” was connected with “chloroplast;“carbon dioxide concentration,“light intensity,’and 


wa mH 


“photosynthesis.’“Light intensity” was connected with “temperature,’“carbon dioxide,"“carbon dioxide concentra- 


nw, 


tion,’ “light energy,’ and “photosynthesis.” “Temperature” was connected with “carbon dioxide,’ “carbon dioxide 
concentration,’ “photosynthesis,” and “light intensity.” Meanwhile, “chlorophyll” and “storage” were not connected 
to other concepts. 

Students in the formal operational stage identified 11 concepts after the initial class (Figure 3e) and 14 concepts 
four weeks later (Figure 3f). The network of formal operational students after the initial class was structured around 
“photosynthesis,"“light energy,’“light intensity,’ and “carbon dioxide.’“Photosynthesis” was connected with “plant” 
and “light energy.’“Light energy” was connected with “light intensity,’“carbon dioxide concentration,’ and “water.” 
“Light intensity” was connected with “carbon dioxide concentration,“light energy,’ and “carbon dioxide.’”“Carbon 
dioxide” was connected with “plant,"“light intensity,’ and “water.” And the network of formal operational students 
four weeks later was structured around “photosynthesis,"“light intensity,’ and “light energy."“Photosynthesis” was 
connected with “temperature,’“light energy,’“carbon dioxide;’“product,’ and “nutrition.”“Light intensity” was con- 
nected with “temperature,” and “carbon dioxide concentration” was connected with “light energy.’“Light energy” 


wn, 


was connected with “water,“light intensity,’“temperature,’ and “photosynthesis.” 
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Photosynthesis 


Eighth-grade students’ concept network for ‘photosynthesis’ after the initial class and four weeks later based 
on cognitive level is shown in Figure 4. And the frequency and eigenvector centrality index of students’ learning 
concepts of ‘photosynthesis’ after the initial class and four weeks later based on cognitive level is shown in Table 5. 

Students in the concrete operational stage identified 31 concepts after the initial class (Figure 4a) and 27 
concepts four weeks later (Figure 4b). The network of concrete operational students after the initial class was 


wu 


structured around “photosynthesis,’ “carbon dioxide,’ and “light energy.’ “Photosynthesis” was connected with 
“plant,’ “chloroplast,” “glucose,” “starch,’ and “light intensity.’ “Carbon dioxide” was connected with “water,” “plant,” 
“oxygen,’ “absorption, “root,” and “stoma.” “Light energy” was connected with “plant,” “photosynthesis,” “carbon 
dioxide/’“water,’ and “light intensity.’ And the network of concrete operational students four weeks later was struc- 


WH, 


tured around “carbon dioxide,’“photosynthesis,’ and “light energy.’“Carbon dioxide” was connected with “water,” 


wn, wu 


“photosynthesis,” “light energy,’ “glucose,” and “oxygen.” “Photosynthesis” was connected with “plant,” “nutrition,” 


“starch,’“glucose,’ and “carbon dioxide.’“Light energy” was connected with “water, 
“carbon dioxide,’ and “oxygen.” 


WH 


photosynthesis,’“absorption,” 


Table 5 
‘Photosynthesis’ Concept Frequency and Eigenvector Centrality based on Cognitive Level 


Concrete Operational Stage Transition Stage Formal Operational Stage 
Concept After Class Four Weeks After Class Four Weeks After Class Four Weeks 
Later Later Later 

Freq. E. Freq. E. Freq. E. Freq. E. Freq. E. Freq. E. 
Photosynthesis 141 .246 112 197 147 .250 123 289 72 .186 58 427 
Carbon dioxide 101 .620 71 571 81 564 53 448 38 549 28 405 
Light energy 95 345 87 424 97 389 77 576 44 513 34 535 
Water 76 614 49 551 55 577 36 395 28 522 21 389 
Glucose 61 .051 60 .185 61 104 43 077 23 .043 20 105 
Oxygen 58 .099 45 147 40 .109 37 076 25 .057 19 088 


* E: Eigenvector centrality index 


Students in the transition stage identified 30 concepts after the initial class (Figure 4c) and 29 concepts four 
weeks later (Figure 4d). The network of transitional students after the initial class was structured around “water,” 
“light energy,’ and “carbon dioxide.’ And the network of transitional students four weeks later was structured around 
“photosynthesis,’“light energy,’ and “carbon dioxide.’ 

Students in the formal operational stage identified 26 concepts after the initial class (Figure 4e) and 24 con- 
cepts four weeks later (Figure 4f). The network of formal operational students after the initial class was structured 
around “photosynthesis,"“light energy,’ and “carbon dioxide.’ And the network of formal operational students four 


wn 


weeks later was structured around “photosynthesis,’“light energy,’ “water,” and “carbon dioxide.’“Photosynthesis” 


wu wu 


was connected with “light energy,’ “plant,” “chloroplast,” and “carbon dioxide.’ “Light energy” was connected with 


wu mH wu mw 


“light intensity,"“photosynthesis,"“water,’“carbon dioxide,“plant,’ and “glucose.”“Water” was connected with “light 


energy, “carbon dioxide concentration,’ “leaf? and “carbon dioxide.’ “Carbon dioxide” was connected with “light 


intensity,’“water,’“photosynthesis,” and “oxygen.” 


670 


https://doi.org/10.33225/jbse/24.23.658 


LEARNING CONCEPT NETWORKS IN THE PHOTOSYNTHESIS BASED ON STUDENTS’ 


COGNITIVE LEVELS 
(PP. 658-678) 


Figure 4 
‘Photosynthesis’ Concept Networks based on Cognitive Level 
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Connection Concept Networks 
Connection Concept Networks between Subtopics 


The connected concepts for each photosynthesis subtopic were analyzed based on cognitive levels (Figure 
5). Among students in the concrete operational stage, 11 concept subtopics were commonly identified after the 


wu wu 


initial class, including “photosynthesis;"“light energy,’“plants,’“chloroplasts,’ and “carbon dioxide” (Figure 5a). Four 
weeks later, 11 concepts, including “photosynthesis,“light energy,”"absorption,’ and “carbon dioxide concentration,” 
were identified (Figure 5b). For students in the concrete operational stage, “carbon dioxide concentration” was only 
connected with ‘environmental factors’ after the initial class, but four weeks later, it was commonly linked to three 
subtopics. “lodine-potassium iodide solution” was linked to‘photosynthesis products’ after the initial class but was 


linked with ‘photosynthesis reactants’ four weeks later. 


Figure 5 
Connection Concept Networks between Subtopics 
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After the initial class, students in the transition stage identified ten connected concepts within the subtop- 
ics, including “photosynthesis,’“glucose,’“carbon dioxide,’ and “plants” (Figure 5c). Four weeks later, ten concepts, 
including “photosynthesis,” “oxygen,’ “glucose,” and “temperature,’ were connected (Figure 5d). Students in the 
transition stage differed from those in the concrete operational stage in that they only connected the concept of 
“carbon dioxide concentration” with ‘environmental factors’ after the initial class and four weeks later. 

Additionally, after the initial class, students in the formal operational stage identified and connected eight con- 
cepts, including “photosynthesis,"“carbon dioxide,;"“glucose,’ and “water, with the subtopics (Figure 5e). Four weeks 
later, ten concepts, including “oxygen,”"production,’and“photosynthesis,’ were connected (Figure 5f). Students in the 
formal operational stage lost concepts related to inquiry activities, such as “bromothymol blue,"“iodine-potassium 
iodide solution,’ and “blue” after four weeks of class. Through this, it can be inferred that it is developed within the 


cognitive structure by focusing on the concept itself rather than inquiry activity-related concepts. 
Connection Concept Networks between Subtopics and Cognitive Levels 


For each photosynthesis subtopic, the concepts connected by students were structured according to cognitive 
level (Figure 6). In the subtopic ‘photosynthesis reactants’ (Figures 6a and b), there were 15 concepts that students 


WH WM, 


at all cognitive levels had in common after the initial class, including “carbon dioxide,’“absorption,’“storage,’ and 
“bromothymol blue.” Four weeks later, there were 12 concepts, including “carbon dioxide,’ “hydrilla verticillata,’ 
“nutrition,” and “oxygen.” After class, students in the concrete operational stage learned three concepts, students 
in the transitional stage learned six concepts, and students in the formal thinking stage learned only one concept. 
After four weeks of class, five concepts were identified by students in the concrete operational stage, students in 
the transitional stage identified four concepts, and one concept was identified by students in the formal opera- 


tional stage. 
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Figure 6 
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students in the concrete operational stage identified one concept, students in the transitional stage identified 
five, and students in the formal operational stage identified three. 

For the subtopic “environmental factors” (Figures 6e and f), the concepts that students of all cognitive levels 
had in common after the initial class and four weeks later were “photosynthesis, “water, “light energy,’ “light in- 
tensity,’“oxygen,’“plant,’ and “carbon dioxide concentration.’ Eleven common concepts were identified after the 
initial class, and ten were identified four weeks later. The difference is that “starch” was commonly identified only 
after the initial class. Meanwhile, after the initial class, students in the concrete operational stage identified five 
concepts, and students in the transition stage identified two. After four weeks of class, students in the concrete 
operational stage identified three concepts, and students in the transition stage identified five, confirming that 
students in the formal operational stage only understood core concepts. In addition, this verified that the higher 


the cognitive level, the more students learned scientific concepts. 
Discussion 


In this study, the concept and connected concept networks were analyzed depending on students’ cogni- 
tive structures after classes based on three photosynthesis subtopics, ‘photosynthesis reactants, ‘photosynthesis 
products, and ‘environmental factors’ The photosynthesis learning concept questionnaires were analyzed twice: 
immediately after the initial class and four weeks later. As a result of the analysis, depending on students’ cognitive 
levels, there was no significant difference in the number and type of learning concepts students acquired after 
class and four weeks later, but there was a difference in the concept network structure. 

Regarding concepts related to photosynthesis, students at all cognitive levels commonly recognized the 
concepts of “photosynthesis,” “plants,” “water,’ “light energy,’ “oxygen,” “carbon dioxide,’ and “glucose” (Figure 5). 
The concepts cover all important topics related to photosynthesis, and the key concepts for each subtopic were 
strongly connected. For example, in the case of “environmental factors,’ it was confirmed that the concepts of 
“light intensity,’“carbon dioxide concentration,’“temperature,’ and “photosynthesis” were robustly linked among 
students at all cognitive levels (Figure 3). 

Meanwhile, students found that their subsequent learning influenced the formation and structuring of con- 
cepts formed by prior learning. For example, after four weeks of class on ‘photosynthesis reactants, the concept of 
“carbon dioxide concentration” was presented. Through this, it was confirmed that the concepts learned through 
subsequent learning were structured together with those of prior learning. 

Students in the concrete operational stage had difficulty differentiating between similar concepts. In the case 
of ‘environmental factors; students initially connected “photosynthesis” with “light energy,’ “carbon dioxide,’ and 
“temperature” but did not make the connection with “light intensity” or “carbon dioxide concentration,’ which are 
similar but more detailed concepts. Even four weeks later, this phenomenon continued. The network seemed to only 
connect the similar concepts of “light energy” and “light intensity.’ In addition, students in the concrete operational 
stage could not clearly distinguish the relationship between concepts. For example, “iodine-potassium iodide solu- 
tion,’ presented as an inquiry activity to check the substances produced from photosynthesis, is recognized as a 
necessary rather than produced substance after four weeks of class. Therefore, it can be inferred that students failed 
to connect concepts and subtopics related to inquiry activities. It is known that students have difficulty learning 
because the content about photosynthesis is an invisible phenomenon, contains abstract elements, and consists of 
connections between complex concepts (Marrnaroti & Galanpoulou, 2006). Accordingly, it is more likely that students 
in the concrete operational stage will experience more difficulties in learning. Therefore, for them, a teaching and 
learning strategy that clearly distinguishes similar terms and chunks them can clarify their relevance (Kim et al., 2023). 

In the formal operational stage, students distinguished concepts more clearly than at other cognitive levels. 
However, they tended to focus more on concepts rather than engaging in inquiry activities. For example, even 
after four weeks of classes on “photosynthesis reactants,’ students were unable to identify concepts related to 
inquiry activities such as “iodine-potassium iodide solution,’“bromothymol blue/’“hydrilla verticillata,” and “blue” 
Although the teachers explained the concept focusing on the inquiry activity (Kim et al., 2023), students in the 
formal operational stage seemed to learn primarily based on the concept, deviating from the teacher's explana- 
tion. This could be attributed to the ease with which students can explain phenomena using scientific concepts 
(Cochran-Smith & Lytle, 1999; Duit & Treagust, 2003). 

In the three photosynthesis subtopics, the concepts that students at all cognitive levels connected were“pho- 
tosynthesis,"“water;“light energy,“oxygen,plants,“carbon dioxide,’and “glucose.’ These concepts correspond with 
‘photosynthesis reactants’ (light energy, water, carbon dioxide) and ‘photosynthesis products’ (oxygen, glucose). This 
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confirms that students’ understanding of photosynthesis focuses on the materials and products required, which 
is believed to be because photosynthesis has already been learned in the ‘Structure and Function of Plants’ unit in 
elementary school (MOE, 2015). And, this indicates that ‘environmental factors, first taught in lower secondary school, 
were not learned properly. To solve scientific problems, students need to understand many concepts. To achieve 
this, students should be able to expand their understanding by connecting their prior concepts, the concepts they 
are currently learning, and the concepts they will learn in the future. However, it is known that students have the 
most difficulty making connections between concepts, which are essential for learning (Ibafhez & Martinez-Aznar, 
2005). To help make these connections, teachers need to present concepts commonly recognized by students of 
all cognitive levels (e.g. “photosynthesis,” “water, “light energy,’ “oxygen, “plant,” “carbon dioxide,’ and “glucose”) 
by structuring them into subtopics. If the curriculum is presented by structuring the content knowledge into core 
concepts, it can help students understand by showing them the systematic structure of biology (Kinchin, 2011). In 
this way, it is thought that it will be helpful in learning concepts related to photosynthesis if subtopics and concepts 
are structured and presented using learning concepts in the taught curriculum. 


Conclusions and Implications 


This study aimed to analyze 8th-grade students’ concepts and connected concept networks based on their 
cognitive level after an initial class on photosynthesis and four weeks later. The conclusions are as follows. First, 
students at all cognitive levels could not clearly distinguish each subtopic’s concepts. Accordingly, they identified 
concepts that were not directly related to subtopics. 

Second, regardless of their cognitive level, students understood the concept of photosynthesis with a focus 
on reactants and products. Regarding the photosynthesis subtopics, the concepts students commonly learned 
after the initial class and four weeks later were “photosynthesis, “water, “light energy,’ “oxygen,” “plants,’ “carbon 
dioxide,’and “glucose.’This is based on what students have learned in the previous curriculum; thus, there is aneed 
to connect and chunk concepts that have already been learned and new concepts. 

Third, depending on students’ cognitive levels, there was no significant difference in the number and type of 
learning concepts they acquired after class and four weeks later, but the concept network structure differed. Students 


mi mm, 


commonly recognized several key concepts related to photosynthesis, including “photosynthesis,“plants;“oxygen,’ 
“water,“light energy,’“carbon dioxide,’ and “glucose.” However, there were differences in how these concepts were 
linked in the concept network. Students in the transition stage showed that concepts related to inquiry activities 
were linked to form a network. Students in the formal operational stage could reorganize their concept network 
by focusing on specific concepts. In addition, students in the formal operational stage not only frequently identi- 
fied these key concepts compared to students at other cognitive levels but also had a high eigenvector index in 
connection with other concepts, occupying a central position in the concept network. 

Fourth, after four weeks, there were differences in the presented concepts and the structure of the concept 
network depending on the cognitive level. For example, concepts learned later were included, and concepts related 
to inquiry activities disappeared. Therefore, due to various factors such as newly learned content and retained infor- 
mation four weeks later, students cognitively restructured their knowledge into a form that was easier to understand. 

Thus, students in the concrete operational stage were less able to clearly distinguish similar concepts when 
learning photosynthesis and related concepts were not properly linked. Additionally, those in the formal opera- 
tional stage tended to engage in cognitive restructuring by focusing on the concept itself. Therefore, teachers 
must clearly distinguish similar concepts by considering students’ cognitive levels and present related concepts in 
chunks. In addition, concepts related to inquiry activities and newly presented concepts must be taught in relation 
to concepts that students already know. 


Limitations of the Study 


This study has some limitations. First, the participants of this study received traditional lecture-based teaching 
in the Republic of Korea. Therefore, in future studies, there is a need to analyze learning concepts in various types 
of teaching methods, such as inquiry activities and discussions. Secondly, a questionnaire was used to analyze the 
learning concepts of these students. This has the limitation that only concepts limited to the questionnaire can 
be presented. To complement this, a more in-depth analysis of the learning concept network can be performed 
through the process of visualizing the concepts presented by students as a network using think-aloud in the 
process of solving problems. 
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Abstract. This study aimed to explore 
the familiarity with and opinions toward 
Education for Sustainable Development 
(ESD) among Montenegrin students at the 
end of their basic school years and upon 
entering upper-secondary school. The 
participants of the study were gathered 
in 2022, and the data were recorded via 
an online survey platform. The results 
were obtained from 705 students in the 
first grade of upper-secondary school. The 
results indicate a notable lack of student 
familiarity with the content of the ESD 
curriculum, despite it being a compulsory 
part of the basic school since 2014. The 
content listed in the curriculum of ESD 

for basic school is only sporadically or 
rarely integrated into teaching practice. 
Respondents mostly showed a neutral 
opinion toward the ESD content and its 
implementation practices. They did not 
demonstrate a clear majority opinion 
toward embracing sustainable thinking 
and behavior. Based on the study findings, 
there is an urgent need for comprehensive 
teacher training in all subjects for basic 
school teachers. The study recommends 
developing effective assessment tools 

for ESD concepts, providing teaching 
materials, and appropriate literature to 
integrate ESD content into each subject, 
ensuring the competent achievement of 
ESD goals. 
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Introduction 


It is easy to see that solutions to many global challenges related to 
unsustainable human activities remain unresolved due to the failure to 
negotiate workable solutions between different stakeholders. Therefore, to 
effectively address the pressing global need for sustainability (Meadows, 
2014), it is crucial not only to advance policies, technologies, and practices 
but also to focus on fostering the action competence of individuals as key 
agents of change (Jensen & Schnack, 2006; Arbuthnott, 2009; UN, 2015). 
However, to be effective and universal, education for sustainability should 
not rely on individual efforts by enthusiastic teachers, but on the efforts of 
entire educational systems. The backbone of education for sustainability at 
the state level is the national curriculum, which should be analyzed from 
different angles, including the students’ perspective. 

Even though adults have significant decision-making power, the influ- 
ence of children who have recently entered basic and secondary schools on 
the acceptance of global sustainable and environmentally friendly practices 
should not be underestimated, as figures such as Greta Thunberg demon- 
strate (Sabherwal et al., 2021; Samuelsson & Kaga, 2008). Today's students 
are the future decision-makers, and it is therefore crucial to educate them 
in such a way that they develop the competencies for effective action when 
needed. While it is impossible to be completely certain about the value of 
the knowledge and skills taught today in the distant future, it is certain that 
at least some of them will be needed to deal with devastating environmen- 
tal problems. Although the specific knowledge and skills taught today may 
evolve as technology advances and environmental challenges emerge, the 
opinions and attitudes towards sustainable living remain a constant that 
should be promoted through the coordinated efforts of the education system 
and other environmentally conscious agents of change. 

Therefore, studies of curriculum components and participants’ experi- 
ences and perspectives at the system-wide level are needed to provide poli- 
cymakers with insights into what works in education at the state or country 
level so that they can implement policies and strategies for better learning 
outcomes in the school system. 
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Education for Sustainable Development 


The need for education for sustainable development stems from the realization that persistent and often un- 
sustainable human activities have led to a range of serious, even catastrophic, environmental problems. Reflecting 
an era in which human activities such as resource exploitation, pollution, urbanization, and the fragmentation of 
natural habitats contribute to climate change, biodiversity loss, and crises, such as uncontrolled migrations, among 
other things, the term ‘Anthropocene’ has been coined to refer to the most recent geological epoch (Steffen et al., 
2011). Living in the Anthropocene, where humans are both causes and victims of environmental problems, makes 
it necessary to find solutions that promote sustainable practices (Ruggerio, 2021). 

Recognizing the urgency of global environmental protection, various international summits have been held 
since the mid-20th century, including the UN Conference on the Human Environment in Stockholm (UN, 1972), the 
UN Conference on Environment and Development in Rio de Janeiro (UN, 1992), the World Summit on Sustainable 
Development in Johannesburg (UN, 2002), the UN Conference on Sustainable Development in Rio de Janeiro (UN, 
2012), and the 2015 UN Sustainable Development Summit in New York (UN, 2015), where the 2030 Agenda for 
Sustainable Development was adopted. The Agenda for Sustainable Development offers a common blueprint for 
peace and prosperity for people and the planet now and in the future. At its center are 17 Sustainable Develop- 
ment Goals (SDGs) with 169 targets that represent an urgent call to action for all countries in a global partnership. 
However, despite global awareness and efforts around the SDGs, the latest evidence suggests that many of the 
proposed SDGs are unlikely to be achieved by 2030 (Weitz et al., 2023). 

Sustainable development can be thought of in three dimensions (pillars): the environment, the economy, and 
society (Berglund & Gericke, 2015; UN, 2015). These three pillars of sustainability are an essential part of Sustainable 
Development Goal 4 - Quality Education (SDG 4). SDG 4 has seven targets and three means of implementation. 
Of the targets of SDG 4, only target 4.7 (Education for Sustainable Development and Global Citizenship) focuses 
on sustainable development. Due to its importance, it is quoted verbatim (UN, 2023): 


“By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, 
including, among others, through education for sustainable development and sustainable lifestyles, human rights, 
gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural 
diversity and of culture contribution to sustainable development.’ 


However, among the proposals to implement SDG 4, there is a lack of actionable suggestions in international 
documents that can be applied by individual teachers to improve education, especially to achieve target 4.7. 
Recognizing that education is an important tool to promote sustainability, a goal of humanity reflected in the 
Sustainable Development Goals (UN, 2015), two main objectives can be attributed to education systems and the 
institutions working within them. The first objective is to bring young people closer to the material, social, cultural, 
and economic environment that surrounds them. Understanding planet Earth as an interconnected ecosystem is 
essential for developing informed solutions that result in sustainability. Solutions targeting local problems caused 
by global, unsustainable practices are curative; preventive measures that address the root causes of these problems 
are often postponed to an indefinite future, which highlights the second objective: teaching the importance of 
proactive change. Therefore, understanding and recognizing the Anthropocene is crucial for developing sustainable 
practices and strategies to mitigate the negative impact of human activities on the planet. It emphasizes the need 
for responsible and informed decision-making to ensure sustainable coexistence with our environment, which is a 
challenge for humanity (Folke et al., 2021). In response to these challenges, the 2023 SDG Summit was convened 
in New York, marking the beginning of a new phase of accelerated progress on the SDGs with high-level political 
guidance for transformative and accelerated action by 2030. In this process, all information about what has worked 
well and what has gone wrong in education should be scrutinized to create a better future. 

Education for Sustainable Development (ESD) can be interpreted in different ways depending on the per- 
spective (Haubrich, 2007). ESD refers to a method of meeting the needs of the present without jeopardizing the 
ability of future generations to meet their own needs. Environmental literacy encompasses five important facets: 
knowledge, awareness, behavior, commitment, and attitude (Jannah et al., 2013; Partanen-Hertell et al., 1999). It 
is an educational approach that aims to equip people with the knowledge, skills, values, and attitudes necessary 
to contribute to a more sustainable and equitable world (UNESCO, 2005). Numerous studies have addressed the 
question of how to implement ESD from early childhood education to higher education, using various approaches 
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to improve learning. These approaches include the use of school gardens or ecological gardens (Tal & Morag, 2009), 
the inclusion of storytelling (Nerantzaki, 2016), the assessment of students’ sustainability competences (Waltner 
et al., 2019), the application of systems thinking through participatory methods (Kioupi & Voulvoulis, 2019), the 
implementation of active learning activities for recycling (Buil-Fabrega et al., 2019), the use of educational games 
(Gandini et al., 2019), the promotion of extracurricular activities focusing on reflective learning (Diaz-lso et al., 2019), 
the introduction of the flipped classroom as an active learning method (Buil-Fabrega et al., 2019), the design of 
effective learning environments (Sinakou et al., 2019), and the promotion of students’ agency (Chen & Liu, 2020), 
among others. 


The Montenegrin School System 


Education is widely recognized as a foundation of social development. In Montenegro, a series of ongoing 
reforms have been undertaken to align the education system with contemporary trends and quality standards. 
These reforms have touched upon all levels of the education system, signifying a commitment to ensuring that 
Montenegrin education remains responsive to the evolving needs of its society. 

Basic school in Montenegro, encompassing both primary (ISCED 1) and lower secondary (ISCED 2) levels, is 
organized as a single-structure system and holds a central place in the nation’s educational landscape. It is intended 
to be both compulsory and freely accessible to all children aged between 6 and 15. This educational phase spans 
nine years and is divided into three distinct cycles. Students with special educational requirements receive their 
basic education in schools and support centers, ensuring that inclusivity remains a core principle. 

Montenegro's public education institutions offer basic nine-year education services through a network of 162 
schools, two educational centers, and three support centers. Additionally, four private schools have been granted 
licenses to operate in the country (Eurydice, 2023a; Eurydice, 2023b). 


Content and Objectives of the Education for Sustainable Development Curriculum in Basic Schools in Montenegro 


During the 27th session held on March 17, 2014, the National Council for Education established a crucial 
initiative known as “Education for Sustainable Development-Cross-Curricular Area” within subject programs for 
Montenegrin nine-year compulsory basic schools. As a premise for its inclusion in the curriculum, it was recognized 
that Education for Sustainable Development is inherently multidisciplinary because no single subject can compre- 
hensively cover all the essential content required to equip students with the knowledge and skills necessary for a 
sustainable future (Cabrilo et al., 2014). 

To implement the cross-curricular model of Education for Sustainable Development, specific topics have 
been defined to promote an integrative approach to general education and create stronger links between subject 
areas. This approach fosters the development of key competences in students. In line with Montenegrin priorities 
and traditions and taking into account international strategic documents in the field of ESD, eight cross-curricular 
topics (CCTs) have been identified. These cross-curricular topics include: 

1. Climate Changes 
Green Economy 
Environmental Protection 
Sustainable Cities and Settlements 
Biodiversity 
Health Education and Upbringing 
Education for and about Human Rights 
Entrepreneurial Learning 


ONAMBWHN 


The Education for Sustainable Development curriculum for basic school presents operational objectives, 
student activities, timetables, and content for each topic and specifies the subjects in which these objectives are 
to be achieved. Overall, it can be stated that the curriculum covers all compulsory subjects that students learn 
during their nine years of basic school (Cabrilo et al., 2014). 
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Research Problem 


As the pursuit of a sustainable future is a fundamental goal for humanity, realizing the moral imperatives 
enshrined in the SDGs requires coordinated action at all levels and demands a transformative change in various 
areas of human endeavor. Education has an undeniable role to play in this ambitious endeavor, serving a dual 
purpose. Quality education is emphasized by the United Nations as one of the 17 SDGs (UN, 2015). Scholars high- 
light its potential as a tool to prepare future professionals who will innovate for a world where economic, social, 
and environmental aspects coexist harmoniously (Guerra, 2017; Sandri et al., 2018; Sivapalan, 2016). According to 
Svanstrom et al. (2018) and UNESCO (2018; 2020), these professionals require key competencies such as systems 
thinking, foresight skills, normative understanding, strategic thinking, collaboration, critical thinking, self-awareness, 
and integrated problem-solving. This perspective aligns with the definition of ESD, which emphasizes that activities 
meeting the needs of current generations must not compromise the needs of future generations (Brundtland, 1987; 
Camioto et al., 2017; Gbededo et al., 2018; Leal Filho et al., 2018). In this context, ESD stands out as a transforma- 
tive process. The role of education in achieving the SDGs is crucial (Kopnina, 2020), as it acts as both a catalyst 
for environmentally conscious human action and a driving force for sustainable development (Pogge, 2004). The 
central role of educators as important catalysts of change (Hattie, 2003; Priestley et al., 2013) and the recognized 
influence of their beliefs (Biesta et al., 2015) are also well-documented. 

Despite decades of commitment, the implementation of environmental education, sustainability education, 
and educational frameworks aligned with the SDGs remains a challenge, not only in Montenegro (Government 
of Montenegro, 2022) but also for the international community as a whole (OECD, 2023). The realization of the 
SDGs, which embody this vision, remains elusive due to various factors (Vladimirova & Le Blanc, 2016). Differing 
viewpoints and resolutions, as highlighted by Annan-Diab and Molinari (2017) and Kopnina (2020), reflect a range 
of perspectives that can sometimes diverge from the experiences of educators, schools, and communities. 

The genesis of this study lies in informal dialogues with teachers, revealing concerns that Education for Sus- 
tainable Development (ESD) in Montenegro is inadequately implemented in nine-year basic schools. Thus, there 
exists a significant gap in understanding the current state of ESD curriculum implementation. Heberlein’s (2012) 
three-pronged approach to environmental problem-solving—technological, systemic, and cognitive—illustrates 
that while changes in environmental practices arise from the first two dimensions, sustainable transformation 
hinges on the cognitive domain. In the Montenegrin educational context, formal environmental education in basic 
and secondary schools effectively addresses technological and systemic aspects. Montenegro boasts a network of 
public schools with well-trained educators, and the curriculum incorporates ESD principles. Another compelling 
reason for this research is the lack of prior examination into the implementation of ESD programs in Montenegro, 
despite their introduction in 2014. The aim of the study is to monitor the implementation of the ESD curriculum in 
basic schools by assessing the familiarity of students who have just completed it, as well as their opinions towards 
it. Additionally, this research aims to evaluate how effectively these programs have been integrated into everyday 
pedagogical practices across subjects. 


Research Aim and Research Questions 


In countries like Montenegro, where research resources are limited, there is a significant lack of foundational 
studies on sustainability education. Therefore, this study aimed to address this gap by analyzing students’ familiar- 
ity with and direct opinions of Education for Sustainable Development (ESD) throughout their nine years of basic 
school. The study also sought to explore their perspectives on the meaning and practices associated with ESD. 

Based on this aim, the following research questions were formulated to guide the study: 

RQ1: Familiarity with CCTs: To what extent and through which channels are basic school students familiar- 
ized with Cross-Curricular Topics (CCTs)? 

RQ2: Integration of ESD content: How frequently is ESD content integrated into basic school lessons? 

RQ3: Students’ opinions towards ESD: What are basic school students’ opinions regarding the integration 
of ESD content into lessons? 

By addressing these questions, valuable insights can be gained into the effectiveness of ESD implementa- 
tion in Montenegrin basic schools. The findings inform evidence-based recommendations for improvements to 
educational authorities. 
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Research Methodology 
General Background 


This study is an integral part of the broader research project titled “Implementation of Education for Sustain- 
able Development in Basic and Upper Secondary Schools in Montenegro.’ For this study, an online survey approach 
was implemented using a questionnaire. The aim of the questionnaire was twofold: a) to collect information on 
Montenegrin basic school students’ familiarity with various topics included in the national curriculum for Educa- 
tion for Sustainable Development, and b) to gather student opinions upon completing basic school regarding the 
curriculum topics and their integration into lessons. Data collection commenced on September 17 and concluded 
on November 17, 2022. It is important to emphasize that student participation in the survey was voluntary and 
anonymous. 

Given that the participants in this study are minors, permission to conduct the research was obtained from 
the Ministry of Education of Montenegro. Additionally, the study’s topic and methodology received approval from 
the University of Montenegro. 


Sample 


The study population comprised former basic-school students now enrolled in the first grade of upper-second- 
ary schools across Montenegro. Data were systematically collected from this population to gain a comprehensive 
understanding of the entire basic school curriculum they had experienced, ensuring that students’ responses did 
not influence their educational trajectory. Collecting data from students in their first year of upper secondary school 
was practical, as their familiarity and opinions provide an overview of their experiences across various basic schools 
(Lang & Sorgo, 2024). This approach diversified the sample, reflecting the broader spectrum of schools attended 
by students in upper secondary classes. According to the statistical reports (Monstat, 2023), a total of 7,097 stu- 
dents were enrolled in the first grade of upper secondary education in 2022/23. school year, distributed across 53 
upper secondary schools throughout Montenegro (see Eurydice 2023a; Eurydice 2023b). Because every student 
potentially had an equal chance to respond to the survey, approximately 400 respondents will guarantee a 95% 
confidence level with a 5% margin of error. In the present study, responses from 705 students were collected, with 
634 students providing a full set of responses, allowing for the assumption of representativeness of the sample, 
albeit with well-known limitations associated with self-selection (Elston, 2021). Socio-demographic data were not 
collected to maintain focus on curriculum topics rather than student differences, ensuring complete anonymity. 


Sampling 


The data were recorded via an online survey platform (1KA.si). Data collection took place between 17 Sep- 
tember and 17 November 2022. At the beginning of the process, emails were sent to all secondary schools in 
Montenegro, explaining the objectives of the study and requesting their support. It was emphasized that the data 
collected in this study will be used solely for academic purposes and possible future publications, which should 
provide insights that can contribute to improving learning outcomes and education for sustainable development. 
It was made clear that participation in the survey was both anonymous and voluntary and that students were free 
to withdraw their participation from the survey at any time. 

To maximize data collection, two reminder letters were sent to schools asking them to encourage as many 
students as possible to participate in the survey. To avoid disrupting regular classes, it was suggested that teach- 
ers share the survey link with their students or conduct the survey during computer science class, where access to 
computers is easy. In this way, in theory, every student had the opportunity to participate in the survey. However, 
the survey has the limitation that the responses are collected from a population that is willing to participate, which 
can introduce a certain degree of bias. 


Instrument and Procedures 
The questionnaire was created to collect data for addressing research questions. To ensure the anonymity of 


students, no personal data or information about their background was requested. This approach is advantageous 
as it encourages more candid responses, while also preventing identification of schools and teachers, thereby 
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facilitating potential improvements in current practices. The first part of the study aimed to assess familiarity with 
the content of the eight cross-curricular topics (CCTs) outlined in the Montenegrin ESD curriculum (refer to Table 
1, Table 2). Students were asked, “How did you learn about the above topics related to sustainable development?” 
They were instructed to select the option that best reflected their level of exposure: (1) | heard about this topic for 
the first time in this questionnaire; (2) This topic was only mentioned in school but not explained; (3) This topic was 
only explained in detail in biology class; (4) This topic was explained in detail in several subjects; (5) | only heard 
about this topic in the media; (6) | informed myself about this topic independently of the available literature. The 
content validity of the response format was ensured through consultations with experts in the field. In Table 2, there 
are 27 themes included in the Education for Sustainable Development (ESD) curriculum for basic school. Students 
were asked: “How often have basic school teachers discussed the following topics in one or more subjects?” They 
were instructed to select anumber corresponding to the frequency using a Likert scale (as used in previous studies, 
e.g., Lang & Sorgo, 2024). The scale offered six options: (1) Never; (2) Very rarely; (3) Rarely; (4) Sometimes; (5) Often; 
(6) Very often. Since the CCTs and themes in the first two tables were directly taken from a syllabus, their content 
validity stands unquestioned and was accepted without further scrutiny. In the case of the ESD contents (Table 2), 
all measures (e.g. Bartlett’s test, KMO test) were performed as precursors to the factor analysis. 

In the second part of the study (Table 3), the students were asked for their opinions on the ESD contents and 
practices in classes during basic school. The 7-point semantic differential scale with bipolar adjectives (Gardner, 
1995) offers pairs of opposing statements and a number scale of 1-7 in between. A rating of 1 indicates complete 
agreement with a positive opinion towards ESD, while a rating of 7 indicates complete agreement with a negative 
opinion towards ESD contents. Close to 1 indicates stronger support or positive opinion towards ESD, while closer 
to 7 indicates stronger disagreement or negative opinion towards ESD content. Theoretically, the design of the 
scale comes from flow theory and was adopted from the study by Sorgo et al. (2018), but to ensure validity, the 
items (adjective pairs) were discussed with the experts in the field. The review of the data matrix shows that the 
Cronbach's alpha of the instrument is .964. 


Data Analysis 


In the first phase after data collection, each variable was analyzed for frequencies, missing data, central 
tendencies, dispersion, and normality distribution (Shapiro-Wilk test). Due to the categorical nature of variables 
presented in Table 1, only frequencies of responses are reported. The response format for the items in Table 2 is 
ordinal, hence the median was chosen among measures of central tendencies. Opinions were assessed using a 
scale based on bipolar adjective pairs. Since responses were provided on a scale from 1 to 7, it was assumed these 
numbers represent intervals on underlying latent variables. Therefore, mean and standard deviation are reported 
alongside median and mode. 

As the instruments were not used in this form previously, an exploratory factor analysis (EFA) was conducted 
using principal axis factoring for factor extraction with direct oblimin rotation to assess the latent structure of 
responses. Parallel analysis was employed to determine the extracted factors. The reliability of the instruments 
and resulting factors from the EFA were evaluated using Cronbach's alpha coefficient, with a cutoff value of .7 set 
for proceeding with the analyses. Statistical analyses were performed using the open-source software Jamovi, 
version 2.3 (Jamovi, 2022) 


Research Results 
Students’ Familiarity with the Cross-curricular Topics (CCTs) of the Basic School Curriculum of ESD 


Students’ familiarity with the eight cross-curricular topics (CCTs) of the basic school curriculum of ESD are 
presented in Table 1, through frequencies of the type of introduction during basic school. 
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Table 1 
Frequencies of the Type of Introduction to Cross-curricular Topics (CCTs) during Basic School 


Familiarization* 


Code CCTs 
1(%)  2(%)  3(%)  4(%)  5(%)  6(%)  Sum3+4 Sum5+6 

Qic Environmental protection iG = — 2 = a an : 
P 10.9 5.2 34.7 37.8 a3 81 73.8 11.4 
Gis: “SbaNat 100 39 442 81 21 23 523 44 
y 14.2 55 62.6 11.5 3 3.3 74.1 3.6 
103 86 72 295 56 94 367 150 
Sr, Teall edbeanen 14.6 12.2 10.2 41.8 79 13.3 52 21.2 
Ga, coments 118 76 131 263 60 58 304 118 
g 16.7 10.8 18.6 27,3 85 8.2 45.9 16.7 
Sy Gaeneeede 120 94 46 286 72 88 332 160 
g ee 17 13.3 65 40.5 10.2 12.5 47 227 
j . 173 147 69 198 77 42 267 119 
Qid Sustainable cities and settlements 245 208 98 28 10.9 5.9 378 16.8 
Sa deeeuseead 286 117 154 61 61 27 215 88 
y 40.5 16.6 21.8 8.6 8.6 3.8 30.4 12.4 
Sh cenenneaienik 299 158 26 109 59 55 135 114 
nee ing 42.4 22.4 3.7 15.4 84 78 19.1 16.5 


Note. (CCTs) Cross-curricular topics; Familiarization*: (1) | heard about this topic for the first time in this questionnaire; (2) This 
topic was only mentioned in school but not explained; (3) This topic was only explained in detail in biology class; (4) This topic 
was explained in detail in several subjects; (5) | only heard about this topic in the media; (6) | informed myself about this topic 
independently of the available literature. The highest values (modes) are bolded. 


When analyzing the data presented in Table 1, several important findings emerge. The most important finding 
is that all cross-curricular topics (CCTs) were absent for at least some of the students, with a range of about 11% 
for environmental protection and about 40% for Green economy and Entrepreneurship learning (column 1). On 
the other side of Table 1 (sum of columns 5 and 6), the frequencies in the last two columns, relating to media and 
literature, show the importance of structured educational approaches to ESD. In most cases, the 10% threshold 
for self-education and private initiative is not reached, and only in four cases is the percentage in the 10% to 15% 
range. The informing but not formative role of formal education is easily recognizable in column 2, where responses 
that the topics were only mentioned but not explained range from around 5% to 22%. 

The aim for all CCTs to be covered across several subjects, promoting a comprehensive understanding of these 
topics and aligning with ESD curriculum guidelines, has not been fully achieved. Only three topics—Health Educa- 
tion, Human Rights, and Environmental Protection—reach a coverage level of approximately 40%. At the lower 
end of the scale, close to 10%, are the topics of Biodiversity and Green economy. The situation can be described as 
slightly better as topics closely linked to the general objectives of biology lessons (Q1c-Environmental protection, 
Qle-Biodiversity) are mainly introduced within this specific subject. Despite the intention to treat all ESD topics in 
a cross-curricular way, students mainly encounter these topics within the biology curriculum. Familiarity with the 
topics can therefore be deduced from the summation of sums 3 (This topic was only explained in detail in biology 
lessons) and 4 (This topic was explained in detail in several subjects), which show that around three-quarters of 
students receive comprehensive information about Environmental protection and Biodiversity and only around 
one fifth about Entrepreneurial learning. It is evident that students are least familiar with the concepts of Entrepre- 
neurial learning and Green economy during their basic school years, more so, about two-fifths of the respondents 
encountered these cross-curricular while completing the questionnaire for this research. 


Inclusion of ESD Themes in the in Basic School Lessons 


The results are presented as frequency and median values of inclusion of ESD themes in basic school lessons 
followed by factor loadings (Table 2). 
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Table 2 
Frequencies, Median Values and Factor Loadings of the Inclusion of ESD Themes in Basic School Lessons 


Factor Factor 


a 


Code ESD themes N 1(%) 2(%) 3(%) 4(%) 5(%) 6(%) 1 2 
The importance of a healthy environment for human 86 54 86 136 133 136 
eae health ms 13.7 8.6 136 216 211 216 a, eee 
91 47 77 117 168 130 
i 9 
Q6ak How can each of us help preserve the environment? 630 jd 75 122 186 402267 ~«206 4 984 
Consequences of improper nutrition and poor physical 95 53 90 135 133 125 
ey activity a 15.1 8.4 143 214 211 198 a. ee 
a ee ‘ ; 86 67 80 153 132 112 
Q6am__ Significance, composition, and sources of air pollution 630 137 106 127 243 mn 178 4 891 
Q6au Flora and fauna of National Parks, internationally pro- 628 93 55 89 149 130 112 4 888 
tected habitats, and protected species in Montenegro 14.8 8.8 142 237 207 178 , 
Q6ag ‘The problem with waste and the importance of recycling 629 = sa o 100 = my 4 862 


15.1 9.2 122 238 216 18 


: . 100 70 95 141 132 92 
Q6ap Ways to reduce pollution from traffic 630 159144 151 224 4 146 4 851 


: 83 58 11 154 166 97 
Q6ac Consequences of climate change 629 13.2 92 13 245 264 154 4.805 


F 86 62 87 169 148 78 
Q6ab Global warming, the greenhouse effect 630 13.7 98 138 268 235 124 4. 804 


Bsa 105 41 89 113 128 152 
Q6az Respect for diversity 628 167 65 442 48 104 242 4 762 


: 90 71 92 163 135 79 
Q6ah Types and importance of renewable energy sources 630 143 «-113«146~=—Oi ACD 4.759 


106 50 82 132 165 96 


Q6aj Ecological problems in Montenegro 631 16.8 79 13 09 26.1 15.2 4 747 
Q6at psec ie of the destruction of rivers, lakes, 627 ne A : 7 7 : - ' 2 4 747 
Q6af - Le of forests and their sustainable 630 . ie he : hes : . ’ i 4 74d 
Q6an Acid rains and their impact 630 uy be ve fp 7 4.730 
Gear The importance of land, its protection 630 ve _ i : . hig 4 725 
Q6ax Rights and obligations in the community 630 4 A i Bes oe He a 4 707 


: f 105 83 117 163 87 75 
Q6ao_ ‘The importance of rational use of natural resources 630 167 13.2 186 25.9 13.8 19 4 668 


oat : 118 62 105 169 106 70 
Q6aq Causes of biodiversity decline 630 18.7 98 167 268 168 414 4 625 


esi ; 103 11 103 156 112 86 
Q6as Ecological importance of mountain areas 631 163 13 163 247 177 ~ «136 4 563 327 


F ape 133 85 107 144 92 69 
Q6al Sources of noise and its impact on human health 630 244 13.5 17 229 146 rv 3 397 ~—-.458 


F : : 136 92 109 159 83 51 
Q6ai Advantages and disadvantages of fossil fuels 630 16 146 173 252 132 34 3 365 ~=—.462 


‘ : 154 89 116 151 69 50 
Q6ae Sustainable agriculture 629 45 144 18.4 4 "1 79 3 157 
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Code ESD themes N  1(%) 2(%) 3(%) 4(%) 5(%) 6(%) x Factor Factor 


1 2 

« ” 187 82 113 123 80 4 
Q6ad “Green economy 626 29 131 181 196 128 65 3 137 
Q6ba European hii advantages and disadvantages of 630 200 72 85 30 71 72 3 707 

Montenegro's entry into the European Union 317 14 #135 206 11.3 ©6114 

: 174 102 124 122 72 37 
Q6aa_ Sustainable development 631 76 162 197 193 14 5.9 3 557 
157 67 103 133 96 74 


Q6ay The difference between sex and gender 630 49 106 163 214 152 147 3 537 


Note. 1-Never; 2-Very rare; 3-Rarely; 4-Sometimes; 5-Often; 6-Very often; 7-median; The highest values representing modes are 
bolded; ‘Principal axis factoring’ extraction method was used in combination with an ‘oblimin’ rotation. 


When analyzing the frequencies of items shown in Table 2, it became clear that the themes intended for 
student education were absent in a range of about 13% to 30%, and only up to 20% of the themes were included 
very frequently. When analyzing the median values shown in Table 2, it became clear that the topics listed in the 
ESD curriculum for basic schools fall into the “sometimes” (20 cases) or“rarely” (7 cases) categories. However, there 
are instances (Q6aa, Q6ad, Q6ae, Q6ay, and Q6ba) where the mode value is 1 (indicating “never”), and only two 
cases (Q6aw, Q6az) have a mode 6 (very often). 

A further analysis of the frequency of inclusion of ESD themes in lessons shows that Cronbach's alpha of the 
instrument is .981. By applying the EFA (Table 2), two highly correlated factors were extracted, which explained 
68.9 % of the variance. The first factor (eigenvalue 14.37; 53.2 % of the explained variance) includes numerous top- 
ics that explain the impact of resource mismanagement in Montenegro and globally, as well as the importance of 
their sustainable use. The second factor (eigenvalue 4.25; 68.9% of the explained variance) includes the concept 
of sustainable development and the green economy, as well as the benefits of Montenegro's accession to the EU. 
The items Q6ai and Qéal are represented in both factors. 


Opinions of the Basic School Leavers toward Greater Inclusion of ESD Content in Classes 


The results of frequencies, measures of central tendencies, and exploratory factor analysis of opinions toward 
greater inclusion of ESD content in the teaching are provided in Table 3. 


Table 3 
Frequencies, Measures of Central Tendencies, and Factor Loadings of the Opinions toward ESD Contents and Practices in Classes 


Factor Factor 


Code Bipolar adjectives N 1 2 3 4 5 6 7 x 1 9 
1 i ee 
oo a! ee eee er 
Lo a ee ee ee ee 
,_ i ee ec 
Q5a__— Useful-Useless 646 - | a ae 7 ce ie 403 1.9 853 
Q5b_ —_Necessary-Needlessly 641 Se hie hs De ie af ie 4.06 1.88  .852 
LL eee 
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Code Bipolar adjectives N 4 2 3 4 5 6 7 f Pastor. recat 


1 2 
7 mere ee ee on 
Q5c _ Interesting-boring 642 ee he ne be - - i 412 1.98  .906 
7 nt oe ee oe 
ee ee ee 
ee as a ee eee a 


Note. All medians have value 4. The highest frequencies (modes) are bolded. The Table is sorted by increasing mean (y). Semantic 
bipolar scale from 1 to 7: a rating of 1-complete agreement with a positive opinion towards ESD, while a rating of 7 indicates 
complete agreement with a negative opinion towards ESD. s-standard deviation. ‘Principal axis factoring’ extraction method was 
used in combination with an‘oblimin’ rotation. Cronbach's alpha = .964 


An analysis of the results in Table 3 reveals several findings. Analyzing the medians and means shows that 
all medians have a value of four, and the means range from 3.90 to 4.20, indicating a neutral position of the two 
measures for the central tendencies. A better insight can be gained by analyzing the frequencies of the responses. 
The distribution of frequencies shows that there are three items at the top of Table 3 whose mean values are slightly 
below the center point, and the number of those who hold extremely opposing opinions is almost equal (Q5k, Q5j, 
Q5l) with regard to the (un)importance of ESD for human health, a healthy environment, and survival on earth. At 
the lower end of the table are opinions where more students think that ESD is boring, difficult to master, unimport- 
ant for further education and that teachers have difficulties in implementing the ESD curriculum. 

When further analyzing the significance of opinions towards ESD content and practices in the classroom by 
applying factorial analysis, two highly correlated factors were extracted, explaining 74.4% of the variance. The factor 
loadings are listed in Table 3. The first factor (eigenvalue 5.41; 45.1% of the explained variance) contains items about 
the interest, necessity and importance of ESD for students’ future, with opinions being mostly neutral or negative. 
The second factor (eigenvalue 3.52; 29.3% of the explained variance) contains three items on the importance of 
ESD for survival on a global level and for the human health, with opinions divided into two extremes. 


Discussion 


At this point, the problem arises that the results of the present study cannot be directly compared with inter- 
national studies, as the Montenegrin ESD curriculum is in some ways unique, even though the idea of ESD is inte- 
grated into many educational systems (e.g. Rauch, 2002). Another problem was that the references that show some 
similarity to the present study mostly report results from studies at the university level and very rarely at the basic 
or secondary school level (e.g. Sorgo & Kamenéek, 2012) and that even when the populations matched, the focus 
is different (e.g. Boeve-de Pauw et al., 2015). Not exactly helpful in categorizing the results is the diversity of school 
systems, where basic school years before diversification in different directions can be of any length, from 4 years 
(e.g. in the Czech Republic or Germany) to 9 years as in Slovenia or Montenegro (see Eurydice websites for details). 

In seeking to answer the research question about familiarity and the channels through which students receive 
information about and are exposed to the curriculum topics, it is apparent that for all of the topics we surveyed, 
at least some students did not receive information about them during their basic school education. Additionally, 
only 5 of the 8 CCTs from the ESD curriculum were implemented in multiple subjects (28-41% of respondents), 
which is in line with the recommended approach (Van den Branden, 2012). From the perspective of these five 
topics (Environmental protection; Health education; Climate change; Human rights education; Sustainable cities 
and settlements), the results are optimistic. However, in three topics (Biodiversity; Green economy; Entrepreneurial 
learning), reasons and limitations that prevent cross-curricular implementation need to be identified. The highest 
familiarity is around three-quarters of the students for the topics of Environmental protection and Biodiversity, 
while the lowest is around one-fifth for Entrepreneurial learning. It is evident that students are the least familiar 
with the concepts of Entrepreneurial learning and the Green economy during their years in basic school. About 
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two-fifths of the respondents encountered these topics for the first time in this questionnaire, which is a concern- 
ing finding. The interpretation of this result can be summarized with the statement by Silajdzi¢ et al. (2015) that 
governmental and educational institutions fail to recognize their role and do not adequately support the develop- 
ment of green entrepreneurship. 

When analyzing the results regarding the frequency of exposure to the detailed list of themes presented in 
Table 2, it became evident that the themes outlined in the ESD curriculum for basic schools predominantly fell into 
the categories of ‘sometimes’ or even‘rarely’ The most concerning finding is that for all of the themes listed, at least 
some students have never encountered the theme as part of the school curriculum. Combined with the finding from 
Table 1 that self-education cannot be relied upon (columns 5 and 6), the results are truly alarming. These findings 
underscore a significant gap between the intended curriculum and its actual implementation, a well-documented 
issue in the literature (e.g., Penuel et al., 2007). In Montenegro, certain ESD themes were consistently overlooked 
or inadequately addressed in basic education. The data clearly shows that the themes listed in the ESD curriculum 
for basic schools were only sporadically or rarely integrated into classroom practice. For example, regarding topics 
such as the European Union, the ‘green economy; and sustainable development, almost a third of students stated 
that they had never encountered these topics during their basic school years. While these results cannot be directly 
compared with international findings due to contextual differences, they affirm the realization that simply includ- 
ing a topic in the curriculum does not guarantee its effective teaching to students (e.g., Sorgo & Kamenéek, 2012). 

Nearly a decade has passed since the mandatory inclusion of these topics in all basic school subjects (Cabrilo 
et al., 2014), so one would expect a comprehensive coverage of these topics. However, based on the research 
design, it is not possible to definitively answer why certain curriculum topics are not included in daily lessons. It 
could be speculated that teachers may not have been sufficiently prepared or willing to teach this content, despite 
its mandatory status. Another assumption is that the overloaded compulsory curriculum in basic school subjects 
likely restricts teachers’ ability to integrate cross-curricular content, including ESD (Kopnina, 2020). To address 
this issue, a careful revision of the compulsory curriculum is suggested to reduce teachers’ workload. Alongside 
this, implementing mandatory retraining programs for teachers across all subjects could better equip them to 
integrate topics like ESD effectively. By creating a more balanced curriculum and providing ongoing professional 
development, teachers would have the time, skills, and flexibility needed to promote a more holistic and sustain- 
able approach to education. 

Understanding students’ opinions on content delivery practices is crucial for predicting future behaviors. How- 
ever, an analysis of students’ opinions reveals a concerning trend: many perceive ESD similar to the documented 
findings for subjects like biology, which is emphasized as dull, lifeless, and boring (Tranter, 2004). This perception 
suggests that the issue lies not in the importance of the topics themselves but rather in how they are presented 
in schools (Kletecki et al., 2023). At this pivotal stage in their education, students ideally should have acquired 
foundational knowledge, developed sustainable habits, and cultivated awareness about their well-being and the 
planet's future. Yet, their somewhat ambivalent attitudes toward the balanced integration of economic, social, 
and environmental development are less than ideal. While some students express positivity, there is a notable lack 
of a clear and resolute stance among young people on the importance of sustainability, considering their future 
roles as academics, engineers, doctors, and parents. This clearly indicates that the current ESD curriculum in basic 
school has not yielded the anticipated results. This is evident from students’ inability to articulate a more positive 
and informed perspective on sustainable thinking and action. This critical mismatch underscores a significant gap 
between the intended impact of the ESD curriculum and students’ actual perceptions. These findings underscore 
the urgent need to enhance students’ awareness of ESD’s significance, not only for personal growth but also for 
global well-being. Specifically, there is a compelling argument to intensify ESD content in upper secondary schools, 
given the less-than-encouraging outcomes observed in basic schools. To effectively bridge this gap, it is essential 
to conduct a thorough reevaluation of the curriculum’s implementation strategy. This reassessment should focus 
on making necessary adjustments aimed at achieving the desired outcomes of enhancing sustainability awareness 
and cultivating a proactive mindset among students. 


Conclusions and Implications 


The study analyses familiarity with and opinions toward ESD of Montenegrin first-grade upper secondary 
school students (fifteen-year-olds). The results indicate that students are insufficiently familiar with the content 
of the ESD curriculum, although it has been mandatory in nine-year basic school since 2014. The data strongly 
suggest that the themes listed in the curriculum framework for ESD in basic school are only sporadically or rarely 
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integrated into teaching practice. Furthermore, the fundamental objective of integrating all topics from the ESD 
program should be implemented cross-curricularly, has not been achieved. Additionally, fifteen-year-olds mostly 
showed a neutral opinion toward the themes of the curriculum and its implementation practices. They did not 
demonstrate a clear majority opinion toward embracing sustainable thinking and behavior. Overall, the results 
emphasize the discrepancy between the planned curriculum and its actual implementation. 

These results make it clear that the introduction of a curriculum is no guarantee that teachers will incorpo- 
rate it appropriately into their lesson plans. To ensure the effective implementation of the ESD goals outlined in 
the curriculum, it is crucial to develop and improve the comprehensive training of teachers of all subjects in basic 
school in Montenegro and develop effective assessment tools to measure students’ knowledge and application 
of ESD concepts. Also, a careful revision of the compulsory curriculum to alleviate teachers’ workload. In addition, 
it is important to create the conditions for smooth implementation, including the provision of teaching materials 
and appropriate literature for the implementation of ESD content for each subject, which would have detailed 
methodological instructions for implementation. Such measures would lay the foundation for a more successful 
implementation of ESD and raise students’ awareness of the central role of sustainable development for humanity. 
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Abstract. The intricacy of real-world 
challenges in project-based geoscience 
learning is complex to assess with a STEM 
approach; hence, research into an effective 
model is necessary to address current 
issues in education. Understanding the 

role of STEM in resolving challenging 
real-world issues requires integrating 

STEM literacy that is appropriate for 
geoscience education. This study examines 
the effectiveness of the Project-Based 
Geoscience Learning (PJBGL) model in 
enhancing STEM literacy. By integrating 
project-based learning with geoscience, 
the aim is to create an interactive and 
immersive learning environment. The study 
seeks to contribute to knowledge about 

the effectiveness of this model compared 

to the conventional model. The aim was 

to evaluate the PJBGL model’s impact 

on students’ literacy, particularly in the 
geosciences. The study utilized a pretest- 
posttest control group design with 33 
students, 19 in the experimental group and 
14 in the control group, collecting STEM 
literacy data using a test instrument and 
analyzing the data with the Mann-Whitney 
test. Effect Size calculation using Cohen's d 
and increased STEM literacy was tested with 
the N-Gain formula. The research results 
show that the PJBGL model is more effective 
in increasing STEM literacy in geoscience 
learning than the conventional learning 
model. This finding holds implications 

for educators and curriculum developers 
seeking innovative strategies to cultivate 
students’ STEM literacy. 
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Introduction 


The geoscience discipline explores Earth’s processes and composition 
through rock and mineral analysis. The field of geoscience focused on the 
study of Earth’s processes and materials through the analysis of rocks and 
minerals, faces considerable educational hurdles (Shirazy et al., 2021). In 
many academic institutions, geoscience is marginalized within the Science 
Education Subfield, resulting ina need for more specialized courses dedicated 
to its study (Guhlincozzi & Cisneros, 2022). This issue is further exacerbated 
by financial constraints related to geoscience practical work, which often 
leads to its neglect at various educational levels (Akhter et al., 2021; Ayeh & 
Bleicher, 2021). Consequently, students’ understanding of fundamental geo- 
science concepts could be improved. Students have misconceptions about 
earthquakes, plate tectonics, weathering, plates floating on the ocean, and 
earthquakes moving (Koehler & Wild, 2018). Karst landscapes largely lack 
misconceptions among students (Francek, 2013). Some students confuse 
hardness with brittleness in geoscience concepts (Giotopoulos et al., 2022). 
Addressing these challenges is crucial for fostering a deeper appreciation 
and comprehension of geoscience, ultimately empowering students to en- 
gage more meaningfully with the field and contribute to its advancement 
(Mogk, 2021). 

Geoscience education encounters several challenges, such as cur- 
riculum misalignment with national standards, insufficient focus on spatial 
visualization skills, and a need for more emphasis on local geological issues 
(Brudzinski et al., 2024). Despite its interdisciplinary nature and reliance on 
direct observation principles, geoscience education faces difficulties due to 
limited resources, which impede its ability to adequately prepare students for 
real-world challenges (Barth-Cohen & Braden, 2022). Considering the critical 
role of geoscience in addressing modern environmental and natural resource 
issues, improving geoscience literacy is crucial for fostering essential skills 
in prediction and decision-making (Rodrigues et al., 2023). To address these 
challenges, it is necessary to bridge the gap between theoretical knowledge 
and practical application (Wang et al., 2024), thereby enhancing the effective- 
ness of geoscience education in preparing students for the complexities of 
the natural world (Vasconcelos & Orion, 2021). 
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Geoscience is characterized by a multidisciplinary nature means that the field of geoscience involves the in- 
tegration of knowledge and methodologies from various scientific disciplines to study the Earth and its processes 
comprehensively (Mulder & Cordani, 2023). Geoscience, also known as Earth science, is an independent scientific 
discipline. It encompasses a broad range of sub-disciplines that study the Earth and its processes (Wang et al., 
2023). This multidisciplinary approach is crucial in understanding the operational mechanisms and evolutionary 
patterns of the Earth system. Geoscience integrates principles from physics, chemistry, biology, and mathematics 
to understand the Earth's structure, composition, and the processes that govern its changes (Power et al., 2023). 
Furthermore, the field of geophysics, within geoscience, emphasizes the importance of publishing studies that 
provide a comprehensive overview and synthesis of the current state of knowledge (Florindo et al., 2023; Mogk & 
Goodwin, 2012; Zheng et al., 2018). Additionally, the relatively young synthetic geographic discipline of geosophy 
also highlights the integration aspect in science, focusing on the interpenetrating unity of mineral, organic, and 
human components in geographical features, showcasing the integration of diverse perspectives and method- 
ologies in studying human space (Lynn, 2000; Martyniuk et al., 2023; Pree, 2017). By integrating knowledge and 
methodologies from various scientific disciplines, geoscientists can gain a more comprehensive understanding 
of the Earth and address complex challenges facing our planet. 

However, there is a recognized need to improve communication and group cooperation skills, which can be 
addressed through innovative approaches like project-based learning (PBL) (Hussein, 2021). Integrating PBL with 
STEM education offers a promising solution to these challenges (Fang et al., 2021). Despite these strengths, empiri- 
cal research specifically focused on STEM literacy development within geoscience education is limited, indicating 
a critical knowledge gap that needs to be addressed (Maric et al., 2023; O’Connell et al., 2021). 

The interconnection between STEM and geoscience is evident in the diverse range of scientific principles, 
technological tools, engineering applications, and mathematical techniques that are applied to study and under- 
stand the Earth system. This interdisciplinary approach is essential for addressing complex challenges related to 
natural hazards, environmental sustainability, resource management, and climate change mitigation Geoscience 
is fundamentally a scientific discipline, encompassing the study of the Earth’s structure, processes, history, and 
interactions between it is various components. Technology plays a crucial role in advancing geoscience research, 
exploration, and monitoring. Remote sensing technologies, such as satellites and drones, provide valuable data 
for mapping and monitoring Earth's surface features, climate patterns, and environmental changes. Engineering 
principles are applied in various aspects of geoscience, including geological engineering, geotechnical engineer- 
ing, and environmental engineering. Mathematics provides the quantitative tools and techniques necessary for 
analyzing, modeling, and interpreting geospatial data and phenomena. 

The specific objective of this study includes evaluating the impact of project-based learning with a STEM 
approach on student literacy in science, technology, engineering, and mathematics (Baran et al. 2021; Rahmania, 
2021). By drawing on existing literature and empirical studies, this research seeks to contribute to the advancement 
of geoscience education by providing evidence-based insights into effective pedagogical practices that enhance 
STEM literacy in this field (Andrews et al., 2022; Bao & Koenig, 2019) 

Building upon previous research on project-based learning and STEM education, this study integrates these 
approaches to assess their effectiveness in enhancing STEM literacy within the geoscience domain (Gallay et al., 
2021; Wieselmann et al., 2022). By employing assessment techniques such as tests, attitude scales, and performance 
evaluations, the study seeks to evaluate students’ literacy in various STEM domains comprehensively (Zhang et al., 
2021). Through empirical observation and analysis, the research aimed to contribute to the advancement of STEM 
literacy development in geoscience education, thereby addressing the identified knowledge gap (McLaughlin & 
Bailey, 2023). 

Based on the results of a study of geoscience curriculum documents in several countries, it can be said that 
the approach and structure of the geoscience curriculum vary in various countries. The British, Canadian and Aus- 
tralian curricula were chosen as references because they have good National Curriculums and provide examples 
of how geoscience can be integrated effectively in overall science subjects. In England, the geoscience curriculum 
is based on the National Curriculum for Science, which places geoscience material in the subject “Science” which 
covers four main areas, namely: Earth and Space, Materials, Energy and Earth and Environment (Balmer, 2019; King, 
2008; King, 2010). Meanwhile, in Canada, the geoscience curriculum varies by province and territory but generally 
refers to national curriculum guidelines that cover major scientific disciplines such as Biology, Chemistry, Physics, 
and Earth and Space Sciences (Metz, 2013; Milford; 2014; Blades, 2019). In Australia, the geoscience curriculum 
refers to the Australian National Curriculum, where geoscience material is covered in the field of “Science” which 
consists of three main disciplines: Earth and Space Sciences, Physical Sciences, and Biological Sciences (Maude, 2014; 


695 


https://doi.org/10.33225/jbse/24.23.694 


Journal of Baltic Science Education, Vol. 23, No. 4, 2024 


IMPROVING STEM LITERACY THROUGH PROJECT-BASED GEOSCIENCE LEARNING (PJBGL) ISSN 1648-3898 | /erint, 


dae ISSN 2538-7138 /oniine/ 


(PP. 694-709) 


Halbert and Salter, 2019; Tomas et al., 2020). Thus, although there are variations in the approach and placement 
of geoscience material in the curriculum in each country, geoscience education is still recognized as an important 
part of science education at the international level. 

STEM literacy refers to the ability to apply concepts from science, technology, engineering, and mathematics 
across different contexts. It plays a crucial role in today’s education system and beyond, especially with the rapid 
growth of STEM fields (Kristin et al., 2023). STEM literacy involves applying STEM concepts in diverse settings. It 
includes problem-solving, knowledge acquisition, communication, decision-making, and integrating science, tech- 
nology, engineering, and mathematics effectively (Teo et al., 2021; Wannapiroon et al., 2021). Research conducted 
(by Agussuryani et al., 2022) produced a literature review that STEM literacy can empower HOTS to increase learn- 
ing activities (Irma & Setiawan, 2023). The study result has shown that applying STEM learning to students’ STEM 
literacy grows pretty well and has a significant impact. According to this study, effective teacher collaboration is 
necessary for STEM learning to exchange knowledge regarding the challenges and solutions students encounter 
when working on project assignments (Wang et al., 2020). STEM literacy is very important for success in modern 
society because it allows individuals to understand and navigate complex issues related to science and technol- 
ogy (Fortus et al., 2022). 

Project-Based Learning (PJBL) has been recognized as an effective pedagogical strategy for improving student 
learning outcomes by fostering deep learning, aligning with 21st-century skills like critical thinking and communi- 
cation (Dan & Xueyan, 2023). PJBL aims to enhance various skills and core literacy, promoting active learning and 
development of abilities (Sun, 2023). PJBL promotes direct learning and encourages students to collaborate, think 
critically, and solve real-world problems (Gabuardi, 2021). Research conducted by Nastiti et al., (2023) produced 
a literature review that PJBL has a significant positive effect on the ability to think creatively. Zhang & Ma, (2023) 
produced a literature review that, compared to traditional teaching models, project-based learning significantly 
improves student learning outcomes and contributes positively to academic achievement, affective attitudes, and 
thinking skills, especially academic achievement and (Maros et al., 2023), proving higher efficiency in teaching us- 
ing project-based learning compared to traditional verbal-visual teaching. The study has shown that Project-Based 
Learning increases student involvement and knowledge retention (Vesikivi et al., 2020). 

Geoscience learning focuses on the study of the process and materials of the Earth (Vasconcelos & Orion, 
2021). This involves understanding concepts such as geology, meteorology, and oceanography (McNeal & Petcovic, 
2020). Research conducted by McLaughlin & Bailey, 2023) produces geospatial research in the future must explore 
the best way to include spatial skills into the classroom for a long period and must focus on the spatial reasoning 
process and strategies used by students when problematic spatial phenomena, especially at the elementary and 
secondary school level (LaDue et al., 2022). They compiled an assessment of the literature from the Geoscience 
Education Community to the model, highlighting the gaps in the field’s current body of knowledge and the potential 
to promote active geoscience education research. So, it is recommended for future active learning investigations 
in geosciences by integrating future geoscience education with active learning models. Geoscience education 
faces challenges such as curriculum mismatch and lack of emphasis on the main skills (Riihimaki & Viskupic, 2020). 
Increasing geoscience literacy is very important to overcome environmental problems and natural resources 
(Vasconcelos & Orion, 2021). 

The development of PJBGL is very important to increase stem literacy in geoscience education. Geoscience 
education faces challenges such as marginalization in the sub-field of scientific education and limited resources 
(Rowan & Dahl, 2023). The PJBGL model integrates project-based learning with a focus on Stem literacy, which aims 
to increase student understanding and access to authentic research opportunities. Further research is needed to 
evaluate the effectiveness of the PJBGL model in increasing STEM literacy and overcoming the challenges faced 
in geoscience education. 

Research on the effectiveness of integrating project-based learning with a focus on STEM literacy in geosci- 
ence education on student understanding, attitude development, and access to authentic research opportunities 
is crucial. While project-based geoscience learning aimed at enhancing student literacy has shown continued 
growth, there needs to be more specialized research focusing on project-based geoscience learning specifically 
targeting STEM literacy. Despite the widespread availability of information and practices related to PJBL, further 
research is needed to determine the success rate of PJBL in enhancing STEM literacy, particularly in the context of 
geosciences. Addressing this challenge requires conducting in-depth research with careful design and adequate 
control measures. This research has aimed to test the effectiveness. The research question was: Was the PJBGL 
model more effective in increasing STEM literacy compared to conventional models? 
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Research Methodology 
General Background 


The research quasi-experimental pretest-posttest non-equivalent control group design follows the meth- 
odological framework outline (Creswel, 2017). The study procedures encompassed several key steps: initially, 
administration of STEM literacy pretests to both the experimental and control group, subtractive implementation 
of the treatment with a focus on stem literacy in the experimental group, and the control group using a conven- 
tions model. The effectiveness of the PJGBL model is tested by comparing the average value of the pretest and 
posttest experimental group and control group. The research was carried out over four meeting sessions at the 
basic physical laboratory of Palangkaraya State Islamic Institute, Central Borneo, Indonesia, in November-December 
2022 with the researcher as the teacher and employee of the PJBGL model. 


Population and Sample 


The research population consists of Physics Education Study Program students at Palangkaraya State Islamic 
Institute, Central Borneo, Indonesia as many as 104 students. This research uses purposive sampling, where the 
sample is selected based on criteria relevant to the research objectives. Purposive sampling was chosen because 
this method allows researchers to selectively choose samples with characteristics or experiences relevant to the 
phenomenon under study, thus allowing researchers to obtain rich and in-depth information related to the re- 
search topic. Thus, using purposive sampling allows researchers to effectively achieve research goals by focusing 
on the most relevant samples and can provide significant insights into the phenomenon under study. This study 
focuses on the quasi-experimental group (19 students) and the control group (14 students) based on the number 
of students in the classroom. The experimental group undergoes the intervention, while the control group will be 
used as a comparison to assess the treatment outcome. The sample has expressed willingness to be the subject 
of the study by being anonymized. 

The demographic condition of the respondents in this study were 4th-semester Physics Education Study Pro- 
gram students aged 18-19 years. Respondents also have the same socio-cultural background, namely students from 
the Central Kalimantan area. In this study, matching was carried out between groups that received treatment and 
groups that were comparable in characteristics. The characteristics that are equalized are age range, grade level, 
and socio-cultural background. These similarly paired groups are then compared, with one being the experimental 
group and the other being the control group. The process of determining the experimental and control groups 
was carried out by matching respondents based on demographic variables such as age range and semester level. 
The experimental and control groups were selected by randomly selecting classes that were equivalent in terms 
of demographics. 


Procedures 
Experimental groups 


In quasi-experimental research with a pretest-posttest design and a non-equivalent control group, the selec- 
tion of experimental classes must be done carefully. One approach that can be used is the PJBGL Model. The class 
receiving treatment or intervention with the PJBGL Model is selected as the experimental group. The selection of the 
experimental class must consider certain characteristics, such as learning ability and motivation, to be comparable 
to the control group (Yin et al., 2021). Thus, differences in results between the two groups can be attributed more 
definitively to the treatment or intervention provided rather than to differences in initial characteristics between 
the groups (Balkin & Lenz, 2021). This step is important to reduce bias and increase the study’s internal validity. 

In general, the syntax of a learning model consists of a structure composed of the main elements and phases 
of the model and how they are put together (Joyce et al., 2016). The PJBGL model is a model development of the 
PJBL syntax (Patton, 2012b) based on Geoscience Learning (Mogk & Goodwin, 2012). The following is the syntax of 
the PJBGL Model syntax: (1) Orientation Phase Fundamental Question. By connecting the body and mind through 
interactions with the natural and social surroundings of the geoscience subject, students can choose/determine 
the theme/topic of the project based on the material/task supplied by the educator; (2) Planning Phase. By follow- 
ing the guidelines for implementing project tasks, choosing activities that can support project tasks, integrating 
multiple options for completing project tasks, planning sources of materials and tools that can help the completion 
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of project tasks, and fostering group cooperation, students can organize the steps of project completion activities 
from start to finish; (3) Exploration Phase. With the lecturer's help, students can create a timeline for carrying out 
their learner's project. The timeline should take into account the activities that have been planned and the amount 
of time that each step of the project requires to be completed. Additionally, students can locate reading materials 
to supplement the project as needed and use maps, sketches, and diagrams to illustrate, validate, rationalize, and 
externalize their understanding of the earth and geosciences. Once students know the assumptions, boundar- 
ies, and uncertainties inherent in the geoscience profession, they can perform their field practicum according 
to norms, language and discourse, tool selection and use, ethics, and values; (4) Monitoring Phase. Students can 
carry out project activities from beginning to end by implementing the project design created and overseen by 
the instructor; (5) Evaluation Phase. Students can create reports and presentations of project outcomes in the 
form of products, such as literary works, artistic creations, or technological works/workshops that are displayed 
as educational product displays to other students, teachers, and members of the community. Students have the 
option to consider their actions as well as the outcomes of their project tasks. Students can share their experiences 
throughout the evaluation stage through discussion to increase performance while completing the project task. 
At this point, comments regarding the procedure and final goods are also made. 

The exploration syntax of the PJBGL model requires students to find or explore with the aim of discovery. 
Students design prototype technology products for geoscience material, fostering intellectual development 
through problem-solving (Hoermann et al., 2022). The monitoring syntax involves routine data collection and 
progress measurement, focusing on process and results through discussion. Students create scientific posters 
to solve geoscience problems, enhancing their ability to understand phenomena from various perspectives and 
generate unique ideas. 

In designing and implementing project-based geoscience learning activities with a focus on rock material, 
increasing the integration of STEM literacy in project-based geoscience learning in each learning syntax facili- 
tates growing STEM literacy. First, in the field of geoscience education, exposure to the latest technology plays 
an important role in improving students’ understanding and application of geological concepts (Hoyer & Hastie, 
2023). in the technological aspect, students are not only introduced but actively engaged in the latest and new- 
est existence in geological mapping and rock analysis data. This hands-on experience serves to bridge theoretical 
knowledge on practical skills, ensuring that students are equipped with both real-world challenges in the field of 
geosciences (Harrell et al., 2023). 

Second, in addition, rock data analysis has been revolutionized by the emergence of advanced technology (He 
et al., 2021). One example of what students do is by analyzing the density and porosity of rocks in Bukit Banama. 
Students use several tools, one of which is the Carbolite Thermo Scientific Furnace used in various laboratory ap- 
plications for the process of heating, drying, and burning materials with accurate and stable temperature control. 
The material used is rock from Bukit Banama. By knowing the density and porosity of the rock, we can gain detailed 
insight into the mineral composition and texture of the rock (C. Zhao et al., 2021). Density is the mass per unit 
volume, while porosity is the ratio of pore volume to total volume (Fu et al., 2021). This information is important 
in evaluating the physical and mechanical properties of rocks, as well as in understanding rock formation and 
transformation processes (Rashid et al., 2022). The density of a rock is affected by the type of minerals and their 
amount, while porosity indicates how much pore space or cracks there are in the rock (Iranfar et al., 2023). Thus, 
knowledge of rock density and porosity is the basis for further analysis of the characteristics of the rock and its 
geological environment (Eyinla et al., 2020). These techniques, when coupled with specialized software, allow 
the quantification and interpretation of rock properties with unprecedented precision (Wang et al., 2024). This 
method is important for obtaining detailed information about the structure and composition of rocks, as well as 
understanding their physical and mechanical properties in more depth (Kahraman et al., 2024). 

Third, in the engineering aspect, students will be involved in designing the project itself and encouraged to 
develop exploration and analysis methods suitable for the project objectives while considering environmental 
factors and resource availability (Beneroso & Robinson, 2022). This process offers real-world experience in apply- 
ing engineering concepts within a geoscience framework, fostering problem-solving skills and creativity (Chen, 
2021). Additionally, in the technical aspect, students will design the project, developing exploration and analysis 
methods tailored to the project’s objectives, considering factors such as environmental impact and resource 
availability (de Rezende et al., 2022). The integration of advanced technology in geoscience education is crucial 
in preparing students for modern geological practice (Chen, 2021). By familiarizing them with the latest tools and 
techniques in geological mapping and rock data analysis, educators can empower students to become proficient 
geoscientists capable of addressing contemporary challenges in the field (Laporte, 2022). The implementation of 
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a Geoscience Project provides students with the opportunity to apply learned concepts and methods in a practical 
setting, enhancing their analysis, research, and presentation skills within a geoscience context (Hoyer & Hastie, 2023). 

Fourth, the engagement with geoscience software and hardware provides students with an opportunity to 
develop critical skills in data manipulation and visualization, which are essential for interpreting complex geologi- 
cal phenomena and communicating findings effectively (de Castro et al., 2024). By immersing students in the use 
of such technologies, educators can cultivate a cohort of geoscientists proficient not only in technical aspects but 
also in communication (Al-Ismaily et al., 2023; Lillico, 2022). Additionally, in the mathematical aspect, students will 
actively engage in quantitative analysis of geological data. They will apply mathematical concepts to interpret data, 
take measurements on a geological scale, and create mathematical models or representations of rock structures 
(Qiu et al., 2023; Shcherbakov et al., 2021). This involvement in mathematical skills will enhance students’ under- 
standing of rocks as entities that can be described mathematically. 

Project-based geoscience learning activities focused on rocks can significantly enhance students’ compre- 
hension and retention of geological concepts. These activities involve hands-on exploration and experimentation, 
allowing students to engage actively with the subject matter and fostering a deeper understanding of geological 
processes. Furthermore, encouraging students to collaborate and communicate effectively promotes the develop- 
ment of essential skills such as teamwork, problem-solving, and critical thinking. Overall, project-based geoscience 
learning activities about rocks deepen students’ understanding of the material and develop multidisciplinary skills 
crucial for a career in the geoscience field. By engaging students in hands-on, collaborative projects, educators 
can cultivate a new generation of geoscientists who are knowledgeable and skilled in applying their knowledge 
creatively and scientifically. 


Control groups 


To maintain the validity of quasi-experimental research with a pretest-posttest design and a non-equivalent 
control group, selecting a control group requires careful consideration. The class that was used as the control 
group did not receive any additional treatment and still underwent conventional learning as usual (Hamilton et 
al., 2021). This class served as a comparison group to evaluate the effectiveness of the intervention given to the 
experimental group. It was important to select a control group comparable to the experimental group regarding 
relevant baseline characteristics, such as learning ability and motivation. This way, differences in outcomes between 
the two groups can be more accurately attributed to the treatment or intervention provided rather than to differ- 
ences in baseline characteristics between the two groups (Zhao et al., 2021). This step is important to reduce bias 
and increase the study's internal validity. 

In an experimental study to assess the physical properties of rocks, two groups of subjects, an experimental 
group, and a control group, were assigned. Both groups were given the same material on the concepts of porosity 
and density of rocks. Porosity refers to the volume fraction of cavities or empty spaces in a rock that can be filled 
by fluid, while density refers to the mass per unit volume of rock. Porosity and density measurements are made 
using standardized methods and instruments, such as digital image analysis or direct mass and volume measure- 
ments. The purpose of providing the same material to both groups was to ensure that their initial understanding 
of the concepts was equivalent before different treatments or interventions were conducted in the experimental 
group. The time spent on these topics for the experimental and control groups was the same duration, namely four 
meetings with 150 minutes per meeting. In the experimental group, the lesson was structured using the PJBGL 
syntax, while the control group used a conventional model with a demonstration syntax. 


Data Collection Instrument 


With ten items designed to evaluate students’ STEM literacy, this study tool comprises questions focused on 
various aspects of STEM literacy. Covering 10 distinct STEM literacy metrics, these questions are designed to gauge 
students’ proficiency in STEM subjects. STEM literacy refers to individuals’ capacity to comprehend and apply the 
principles of science, technology, engineering, and mathematics to address issues, make choices, and engage in 
situations related to STEM. The assessment tool used in this study aims to evaluate the literacy level of participants 
through inquiries covering key areas of science, technology, engineering, and mathematics. The literacy measure 
employed is presented in Table 1. 
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Table 1 
Grid of STEM Literacy Instrument Based on Geoscience Project 


Discipline Indicator 


Given a picture of rock magnetism, students can explain the problem based on the picture correctly. 


Presented data about the results of rock observations, students can evaluate problems based on these data 
appropriately. 


Science 
Presented pictures about the results of observations about geoscience; students can evaluate problems based 


on the data correctly. 


Given a picture of technological principles, students can explain problems based on these technological 
principles appropriately. 


Given the results of data processing about rock magnetism, students can analyze the use of technology ap- 
Technology propriately. 
Given a narrative about rock magnetism, students can formulate problems based on the narrative correctly. 


Given a problem regarding practicum tools, students can develop solutions by using these tools properly. 


Engineering Given a mathematical problem of rock magnetism, students can formulate the problem mathematically cor- 
rectly. 


Presented image data about rock stratigraphy; students can interpret data based on this data correctly. 


Mathematics Given a mathematical problem of rock magnetism, students can interpret the solution accompanied by empiri- 
cal evidence using mathematics about rock magnetism correctly. 


Table 1 presents the results of a comprehensive synthesis of existing literature or expert evaluations, provid- 
ing a detailed breakdown of the key aspects of STEM literacy that were identified as crucial for assessment in this 
study. The table serves as a valuable reference for understanding the specific topics and concepts within science, 
technology, engineering, and mathematics that were considered essential for measuring literacy levels among 
participants. 

The content validity test obtained an average Aiken coefficient value of .99, falling within the good category. 
The construct validity analysis based on the results of the confirmatory factor analysis revealed a p-value of .15, GF 
of .95, AGFI of .92, and RMSEA of .036. The squared error is approximately .036, which is less than .08, and the chi- 
square test yielded a value of 1.19, which is less than 2, with an appropriate index of .95, exceeding the threshold of 
.90. This model, selected from field data, can be applied to various measurements. The Composite Reliability value of 
.7 falls within the reliable category. As the developed instrument has met the validity and reliability requirements, 
it can be utilized by geoscience educators in higher education to measure the STEM literacy of physics students. 

The content validity test obtained an average of Aiken coefficient value .99 in a good category. The construct 
validity analysis based on the results of the confirmatory factor analysis, it was found that p .15, GFI .95, AGFI .92, and 
RMSEA .036. The squared error is approximately 0.036; 0.08, and the chi-square test is 1.19; 2, with an appropriate 
index of .95; .90. This is a model selected from field data that can be used for many different measurements. The 
Composite reliability value of 0.7 is in the reliable category. Since the developed instrument has met the validity 
and reliability requirements, geoscience teachers in higher education can use it to measure the STEM literacy of 
physics students. 

Throughout the application process, this STEM literacy tool has undergone multiple examinations, including 
testing, measurement, validation, and qualitative analysis. As previously said, experts examine and conduct experi- 
ments to test the model. The results of the expert model evaluation according to the preliminary design review, 
the proposed model appropriately emphasizes STEM literacy. An instrument is considered valid if the expert feels 
it can measure the intended variables (Cantabrana et al., 2019; Garg et al., 2021; Howells et al., 2020). The study 
results indicate that out of the 10 items created, they possess a loading factor value. The metrics employed are 
RMSEA and (GFI) (Lukman et al., 2021). The GFI standards have been met as the produced value falls within the 
specified range. This indicates that the model acquired is suitable, suggesting that the instrument's design is sound 
and may be used to assess the distribution of this measurement model. 
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Data Analysis 


The Levene test was utilized to examine the homogeneity of variances. The data is deemed homogeneous if 
the p-value is greater than or equal to .05 (p = .05). The outcomes of these preliminary analysis tests are presented 
in Table 2. 


Table 2 
Analysis Prerequisite Test 


Testing Kind Test Significance Decision 


Homogeneity Levene STEM Literacy: .141 Homogeneous 


The post-test and pre-test data were meticulously analyzed using descriptive and inferential statistical methods. 
Descriptive analysis calculates the mean, median, mode, range, variance, and standard deviation. These metrics 
provide a comprehensive summary of the data distribution and central tendency. Inferential analysis, on the other 
hand, was conducted using the Mann-Whitney test. This statistical test was employed to determine if significant 
differences existed between the effects of the Project Geoscience learning model and the conventional model on 
enhancing student STEM literacy. Effect Size calculation using Cohen's d and the level of increase in STEM literacy 
was tested with the N-Gain formula. This approach allowed for a rigorous evaluation of the effectiveness of the 
different teaching models in promoting STEM literacy among students. 


Research Results 


The descriptive statistical analysis of pre-test data on STEM literacy is encapsulated in Table 3, elucidating 
the baseline proficiency levels before the intervention. Complementing this initial assessment, Table 3 delineates 
the descriptive statistics derived from the post-test data, furnishing insights into the participants’ STEM literacy 
after the implemented measures. Descriptive statistical results Pre-test data literacy STEM is presented in Table 3. 


Table 3 
Descriptive Statistical of Pre-Test and Post-Test STEM Literacy 


Pre-Test Post-Test 
Group N = = N-Gain 
X SD zx SD 
Experiment 19 41.21 1.513 88.21 0.819 a7 
Control 14 40.29 1.619 69.29 1.563 , 


Table 3 shows asummary of descriptive statistical analysis results, indicating that the average posttest literacy 
STEM from the implementation of the experimental class PJGBL was higher than the average STEM literacy from 
the control class. The standard deviation of STEM literacy in PJGBL learning was smaller than the standard deviation 
of the control class. This indicated that the data tended to be convergent towards the mean. 

The Mann-Whitney test was also used to determine the effect of PJGBL on STEM literacy. The following are 
the results of hypothesis testing using the Mann-Whitney Test analysis which are shown in Table 4. 
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Test Statistics Uji Mann-Whitney 


STEM Literacy 


Mann-Whitney 19.500 
Wilcoxon W 124.500 
Z 4.147 
Asymp. Sig. (2-tailed) .0001 


Table 4 shows the Test Statistics output in the Mann-Whitney test. It is known that the Asymp. Sig. (2-tailed) 
of < .001 is smaller than the probability value of .05. Therefore, as is the basis for making the Mann-Whitney test 
decision, it can be concluded that H, is accepted. Thus, it can be said that there is a significant difference in the 
median between the experimental and control classes. This indicates that using PJBGL in the classroom has a 
more significant impact on raising student literacy. A coefficient of 3.81 in the high category was found by ap- 
plying Cohen’s D method to calculate the Effect Size. This demonstrates how well the PJBGL Model improves 
STEM literacy—the outcome of the N-Gain analysis is the following analysis. Following computation, an N-Gain 
value of .79 in the high category was achieved. Based on these values, the PJBGL strategy can raise STEM literacy 
with a very high category. 


Discussion 


The research results revealed a significant difference in increasing STEM literacy between the experimental 
and control classes. The experimental class, which underwent treatment with the PJBGL model, showed a much 
greater increase in STEM literacy than the control class which applied the conventional learning model. These 
findings highlight the significant potential of the PJBGL model in increasing STEM literacy. In this context, the 
differences in results between the two groups suggest that learning designs that focus on projects and STEM 
literacy simultaneously can provide greater benefits than conventional learning methods that do not explicitly 
emphasize aspects of STEM literacy. 

The research findings from various studies on STEM-integrated PjBL models underscore the substantial 
potential of this approach in enhancing STEM literacy (Nurhayati et al., 2023; Nuraini, 2023). A comparison be- 
tween experimental and control groups reveals that learning models combining project-based learning with a 
focus on STEM literacy yield more significant benefits than traditional methods that do not explicitly emphasize 
STEM literacy aspects (Hayuana et al., 2023). These results indicate that integrating projects and STEM literacy 
in educational designs can lead to improved outcomes in terms of students’ motivation, scientific literacy skills, 
and overall learning achievements (Muskania et al., 2023). Therefore, the evidence suggests that educational 
strategies emphasizing both projects and STEM literacy concurrently offer a more effective approach to enhanc- 
ing students’ understanding and proficiency in STEM literacy compared to the conventional learning model. 

Research shows that there is an increase in STEM literacy in classes that use PJBGL. This is because each 
stage or syntax in the PJBGL model is designed to specifically grow students’ STEM literacy abilities. For example, 
project activities in PJBGL not only teach geoscience concepts, but also engage them in problem-solving, 
critical thinking, and communicating effectively in a STEM context. As a result, students who engage in PJBGL- 
based learning tend to develop their STEM literacy skills better than students who learn through conventional 
approaches. PJBGL offers an immersive and comprehensive learning environment where students can gain a 
deeper understanding of STEM literacy. 

Based on the research findings from various studies, it is evident that Project-based Learning with a STEM 
focus can significantly enhance students’ literacy skills in these areas. Studies have shown that interactive STEM 
learning is particularly effective in improving mathematical literacy skills among elementary school students in 
urban areas (Susanta et al., 2023). Additionally, the use of STEM-integrated Physics Digital Teaching Material (PDTM) 
has been found to positively impact students’ conceptual understanding and new literacy skills, including data, 
technology, and human literacy (Susanta et al., 2023). Furthermore, Design-Based Learning (DBL) approaches 
have been shown to improve content learning and foster STEM identity among students, with different DBL 
methods yielding varying outcomes in terms of STEM literacy development (Asrizal et al., 2023). These findings 
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collectively support the idea that PJBGL-based learning can indeed facilitate the development of students’ STEM 
literacy skills more effectively than the conventional model. 

The PJBGL model, which integrates project-based learning with a focus on STEM literacy in geosciences, has 
shown the ability to increase STEM literacy. Modification of the PJBGL syntax involves students in orienting to 
science and technology development in geosciences, integrating this with learning geosciences in the field with 
a STEM literacy approach. The modified syntax, named PJBGL with STEM literacy, consists of five phases: Orienta- 
tion, Planning, Exploration, Monitoring, and Evaluation. The orientation phase facilitates students in selecting 
project themes based on given tasks, integrating mind-body interaction in the natural and social environment 
to build knowledge and attitudes. 

The PJBGL model is more effective than the conventional model commonly applied previously to increase 
STEM literacy. This is because the syntax in the learning model has the advantage of facilitating an increase in 
STEM literacy indicators (Lina & Amidi, 2023; Muzana et al., 2021). Data collection syntax, solution design, and 
communication are the strengths of this model in increasing the ability of STEM literacy (Muzana et al., 2021; 
Purwaningsih et al., 2020). Various ways can be done to increase STEM literacy, namely learning and assessment 
(Hikmawati et al., 2020; Tati et al., 2017; Xu et al., 2019). Ideally, the evaluation activities conducted should align 
with the instructional method employed. The literacy evaluation of STEM will be suitable if the learning method 
utilized already includes that skill. Nevertheless, if the education conducted does not promote these skills, it 
does not imply that STEM literacy cannot be achieved. 

In the PJBGL teaching model, students are helped to solve problems through activity-based learning, 
discovery centered on students (Jackson et al., 2021; McDonald, 2016; Nurlaely et al., 2017). The PJBGL learning 
model encourages students’ science attitudes and encourages them to actively participate in the search for 
ideas and link their experiences with the material being studied. Therefore, Literacy STEM combines science 
and mathematics lessons with elements of technology and engineering in geosciences to attract students. This 
integration is appropriate because students receive the initial exposure to technology and techniques when 
studying geoscience material. This is also shown by the PJBGL model and Literacy STEM, which correlate, namely 
(1) Contextual relevance: The PJBGL model allows students to learn about geosciences in the context of real situa- 
tions or concrete projects. Students can study natural phenomena, environmental problems, or other geoscience 
challenges that are relevant to everyday life. (2) Practical Experience: The PJBGL model allows students to take 
part in geoscience projects that involve field observation, data analysis, and real geoscience problem-solving. 
This helps students understand geoscience concepts more deeply. (3) Active involvement: In the PJBGL model, 
students are actively involved in the planning, implementation, and evaluation of their projects. Students play 
an important role in learning, which increases motivation and understanding. (4) Collaboration and communi- 
cation: Projects in the PJBGL model often involve teamwork and collaboration, which are important skills in the 
fields of geosciences and STEM in general. 

Additional research refers to empirical investigations that provide a more detailed description of the 
processes involved in construction or improvement. This study can clarify the procedure and formation of the 
models created. Firstly, this method enables the establishment to create a fresh framework with a suitable level 
of clarity to concentrate on several issues. Furthermore, users, specifically instructors and students, may readily 
understand the model outcomes. This study presents a model that has been demonstrated to be valid and can 
be utilized by instructors who are closely connected to the context, material, and students. 

The PJBGL learning model received a very positive response from the lecturer and students because the 
range of values of students and educators was in the third range of quartile and maximum grades. This PJBGL 
learning model meets the practical criteria of a learning model, namely (1) the model can be applied in learning 
geoscience material; (2) the time, energy, and cost of using affordable models; and (3) the syntax of the PJBGL 
learning model is by this learning objective, namely, to increase STEM literacy. 

This PJGBL model further encouraged and provided opportunities for students to do foreign tasks and was 
able to hone their ability to explore knowledge with STEM literacy compared to educators who only provided 
routine tasks (Mulyani, 2019; Roshayanti et al., 2022; Satria, 2016). The importance of the role of lecturers in 
facilitating the literacy of students’ STEM must also be in line with the literacy of the lecturer STEM itself. Educa- 
tors are role models in exploring information in science (Cayirdag, 2017; Hines et al., 2019; Lukman et al., 2021). 
Enhancing learning patterns in various areas impacts educators’ mindsets and improves learning patterns in 
educational institutions, as well as the support from families and the surrounding environment. Educators should 
also be able to understand and implement appropriate learning models for their students. 
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Although the PJGBL model has several advantages, this research certainly has limitations, such as project 
models developed too specifically with geoscience material characteristics. Therefore, it is recommended for 
further researchers to develop a project learning model that is more flexible to other materials. 

One limitation of this study is the reliance on one statistical analysis tool, SPSS. While SPSS is powerful and 
widely used, its limitations in handling complex analyses may affect the depth of the findings. The use of Cohen's 
d-effect size test may provide a quantitative measure of treatment effect but may need to capture the nuances 
of treatment effectiveness fully. Future research could explore alternative statistical analysis tools and effect sizes 
for a more comprehensive analysis of treatment impact to overcome these limitations. 

This research demonstrates the effectiveness of the PJBGL model in enhancing students’ STEM literacy fields. 
By comparing an experimental group utilizing PJBGL with a control group employing conventional methods, the 
study underscores the substantial advantage of PJBGL in promoting STEM literacy, corroborating prior research 
on the potential of PJBGL-based approaches to improve students’ STEM literacy competencies. Moreover, insights 
from various studies emphasize the benefits of integrating projects and STEM literacy into educational designs, 
as such approaches have been shown to positively impact students’ motivation, scientific literacy abilities, and 
overall academic achievements. Additionally, research suggests that interactive STEM learning, the utilization 
of STEM-integrated teaching materials, and Design-Based Learning approaches can significantly contribute to 
the enhancement of students’ STEM literacy proficiency. 


Conclusions and Implications 


After conducting comprehensive data analysis and comprehensive interpretation of results, the findings 
of this study provide empirical evidence that convinced that: Based on the results of the discussion, it can be 
concluded that the PJBGL model is more effective in increasing STEM literacy than conventional learning model. 
This study presents a model that has been demonstrated to be valid and can be utilized by instructors who 
are closely connected to the context, material, and students. Research shows that the experimental class that 
received treatment with the PJBGL model experienced a significant increase in STEM literacy compared to the 
control class that applied the conventional learning model. These findings are supported by previous research 
which also highlights the great potential of the PJBGL approach in improving students’ STEM literacy skills. The 
integration of projects and STEM literacy in the design of the PJBGL model has been proven to provide greater 
benefits than conventional learning methods that do not explicitly emphasize aspects of STEM literacy. These 
conclusions demonstrate the relevance and importance of innovative approaches in education to prepare stu- 
dents to face the demands of a world increasingly focused on STEM. This finding holds implications for educators 
and curriculum developers seeking innovative strategies to cultivate students’ literacy skills in STEM disciplines, 
ultimately preparing them for success in an increasingly STEM-centered world. 
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Abstract. The significance of environmental 
education has become more evident, driven 
by global concerns surrounding climate 
change. In Vietnam, a country facing 
significant environmental challenges, 
developing environmental knowledge of 
elementary school teachers is crucial. This 
study aimed to validate a questionnaire 

for assessing professional knowledge of 
elementary school teachers in teaching 
environmental issues, originally designed in 
the Korean context. Recognizing the need for 
adaptation to the Vietnamese educational 
system, language, and local environmental 
issues, the initial 39-item questionnaire 
underwent a validation process. A total of 
86 elementary school teachers participated 
in the construct validity and reliability, while 
eight qualified professionals evaluated the 
content validity. The results indicated that 
content validity indices for each item ranged 
from .88 to 1.00, and a scale-content validity 
index was .90. Exploratory factor analysis 
confirmed construct validity, organizing 

the 27 items into seven dimensions. The 
Cronbach's a coefficient was .86, which 
confirmed the questionnaire demonstrated 
ahigh level of reliability. Therefore, it isa 
reliable instrument for assessing elementary 
school teachers’ professional knowledge 

in teaching environmental issues, which 

is relevant in the Vietnamese context. 

The validated instrument supports the 
development of professional development 
programs and contributes to the broader 
goal of enhancing environmental education 
in Vietnam. 
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Introduction 


The urgency of addressing environmental issues through education 
has become more evident. Education plays a vital role in equipping the 
next generation with the knowledge and skills needed to tackle pressing 
environmental challenges. As societies increasingly recognize the intercon- 
nectedness of human actions and environmental health, it becomes essential 
for educational systems to integrate comprehensive environmental educa- 
tion. Such integration addresses the global concerns about climate change, 
resource depletion, and ecological deterioration, as well as fosters a deeper 
understanding and commitment to sustainable practices among students, 
who are future stewards of the planet (Taboada-Gonzalez & Aguilar-Virgen, 
2024; Varela-Candamio et al., 2018). 

Within the Vietnamese setting, a nation characterized by its remarkable 
natural landscapes and cultural diversity, there recently emerged a compel- 
ling concern about environmental challenges (Chau et al., 2020; World Bank, 
2022). These issues involve a range of complex obstacles, such as deforesta- 
tion, pollution, and the inexorable advancement of urbanization (Hoang et 
al., 2022; Ho et al., 2021). Vietnam is currently experiencing significant impacts 
from climate change and natural disasters, resulting in serious consequences 
for the health of the country’s citizens (Huong et al., 2022; World Bank, 2022). 
Considering the ways to reduce the risks associated with natural environmen- 
tal contamination, it is crucial for Vietnam to prioritize the development of 
environmental knowledge among all its citizens, with a special emphasis on 
elementary students (Hoang & Kato, 2016; Mashaba et al., 2022). Students 
are coming of age in an era characterized by human-induced global warm- 
ing and grave environmental threats (Heck, 2015). Thus, they hold a pivotal 
role as the future generation of a nation, exerting considerable influence on 
sustainable development in society. Their attitudes, actions, and decisions 
will define their nations’ long-term prosperity and ecological sustainability 
(Tayci & Uysal, 2012). 

The purpose of enhancing environmental knowledge among elemen- 
tary students emphasizes the role of elementary teachers as facilitators of 
environmental education (Sukma et al., 2020). The effectiveness of elementary 
teachers is crucial because they establish the foundation for a child’s lifelong 
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learning (Heck, 2015; Hoang & Kato, 2016). Elementary school teachers are critical to developing students’ intel- 
ligence and providing them with the understanding to build a sustainable life (Timm & Bartha, 2021). Therefore, it 
becomes paramount to equip elementary school teachers with the professional knowledge essential for effectively 
imparting environmental education to their students. 

Hence, it is important to have a reliable and valid instrument for evaluating the professional knowledge of 
elementary school teachers in Vietnam. This guarantees that elementary school teachers across Vietnam are well- 
prepared and possess the requisite professional knowledge. Some studies in the literature have confirmed the 
relevance of elementary school teachers’ competencies to content knowledge and pedagogical content knowledge 
in the Vietnamese context (Nguyen, 2001; Thao et al., 2022). However, the existing evaluation instruments may 
not comprehensively address the distinctive challenges and demands of the Vietnamese elementary education 
landscape. While evaluating existing international literature, this study encountered a questionnaire by Lee et al. 
(2018) related to teacher knowledge for environmental education in elementary schools, specifically designed 
in the context of Korea. Notably, an interdisciplinary approach to environmental education is observed to be a 
commonality between Korea and Vietnam. Encouraged by this alignment, this study aspires to apply the Korean 
questionnaire to assess teacher knowledge within the Vietnamese context. 

Education in Vietnam is undergoing ongoing reforms, including national curriculum, textbooks, educational 
approaches, and assessments (Nguyen et al., 2022). This requires teachers to not only possess pedagogical skills 
but also an insight into teaching content, contexts, classroom management, learning objectives, and the flexibility 
to adapt to their students’ ever-changing needs (Pham et al., 2023). As a result, validating an instrument developed 
to measure elementary school teachers’ professional knowledge for teaching environmental issues in Vietnam is 
worthwhile. This study aimed to close the gap between the Korean original questionnaire and the Vietnamese ver- 
sion, as well as ensure its suitability, reliability, and validity in measuring elementary school teachers’ professional 
knowledge in teaching environmental issues in Vietnamese elementary education. 


Theoretical Framework: Professional Knowledge 


The concept of “professional knowledge” reflects a complicated foundation that teachers need in order to 
effectively engage in the process of teaching (Fischer et al., 2012). In 1986, Shulman published the first theoretical 
framework that is extremely valuable in enhancing the professional knowledge of teachers (Neumann et al., 2019). 
Shulman initiated the exploration of teacher professional knowledge, identifying various knowledge bases like 
content and general pedagogical knowledge (Neumann et al., 2019; Shulman, 19876). Subsequently, in 1987, the 
framework was broadened by including seven essential elements. These include content knowledge (CK); general 
pedagogical knowledge (GPK); curriculum knowledge (CmK); pedagogical content knowledge (PCK); learners 
and their characteristics knowledge (LCK); educational contexts knowledge; and educational ends, purposes, and 
values knowledge (PVK). 

In their attempts to model professional knowledge, scholars have reached differing opinions regarding these 
seven components. Grossman (1990) presented four components: knowledge about students; knowledge about 
curriculum; knowledge about teaching strategies; and beliefs about the purposes for teaching a subject at different 
grades. Teaching is the act of combining content knowledge, context knowledge, and pedagogical knowledge 
(Gess-Newsome, 1999). Professional knowledge includes four components: general misconceptions of students, 
curriculum-specific knowledge, teaching strategies knowledge, and teaching objectives knowledge (Carlsen, 1999). 
Furthermore, knowledge of the subject area, understanding of the context and goals of instruction, strategies for 
representing ideas in the classroom, and familiarity with students are all components of professional knowledge 
(Jang et al., 2009; Tuan et al., 2000). Moreover, assessment knowledge is an important element of teachers’ profes- 
sional knowledge (De Jong, 2009). 

Until now, there has been ongoing discourse regarding what constitutes professional knowledge and how to 
assess and integrate it (Young & Muller, 2010). This study adopts the comprehensive view of professional knowledge 
developed by Lee et al. (2018) as teachers’ professional knowledge in environmental education, including seven 
components: (1) content knowledge; (2) general pedagogical knowledge; (3) curriculum knowledge; (4) learners 
and their characteristics knowledge; (5) educational contexts knowledge; (6) educational ends, purposes and values 
knowledge; and (7) assessment knowledge. 
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Measuring Professional Knowledge in the Field of Environmental Education 


In international literature, there is evidence that professional knowledge has been explored across various 
fields such as mathematics, science, history, and literature (Mishra, 2020; Neumann et al., 2019; Tuithof et al., 2021; 
Van Sledright, 2014). However, when it comes to the field of environmental education, the research landscape 
is relatively limited (Lee et al., 2018; Yolcu et al., 2022). Notably, most studies in environmental education have 
primarily focused on the secondary education level (Abdullah & Halim, 2012). 

Within the Vietnamese educational context, it is important to highlight that there is no distinct curriculum 
to instruct environmental education for elementary school students (Ministry of Education and Training [MOET], 
2018). Consequently, teachers are tasked with the challenging role of integrating environmental concerns—cov- 
ering aspects like natural resource conservation, ecosystems, habitat preservation, and pollution—into various 
subjects (Tran et al., 2020). This integration approach often results in the professional knowledge required for 
effective environmental education being overlooked (Danh, 2021). 

Nevertheless, there have been some studies concerning teachers’ knowledge of environmental education. For 
example, the work of Nguyen (2001) indicated that Vietnamese elementary teachers not only lack comprehensive 
knowledge of environmental issues, conservation, and environmental education but also struggle to apply theoreti- 
cal knowledge to their local contexts. While there have been studies addressing teacher competencies, especially 
related to two kinds of knowledge (content and pedagogy), most of these focus on secondary or higher education 
levels (Kieu, 2016; Nguyen, 2018). Moreover, these studies tend to lack coverage of all seven components outlined 
in Shulman’s (1987) or Lee et al’s (2018) frameworks of professional knowledge. 

Furthermore, it was mentioned that interviews and observations are the main methods used to evaluate 
teachers’ professional knowledge. One common limitation observed in these studies is the lack of developed 
assessment tools designed to evaluate teachers’ knowledge of teaching environmental issues, particularly at the 
elementary level (Thao et al., 2022). 


Research Aim and Research Questions 


The field of teacher knowledge, particularly professional knowledge, has been extensively explored in the exist- 
ing body of literature (Chan & Yung, 2018). While there has been a growing emphasis on teacher knowledge in various 
research studies conducted in Vietnam, there is a noticeable lack of attention given to the field of environmental 
education. Based on the literature review, several previously validated instruments were examined, leading to the 
selection of the Elementary School Teachers’ Environmental Pedagogical Content Knowledge questionnaire (Lee 
et al., 2018) to evaluate the teacher's professional knowledge regarding teaching environmental issues in Vietnam. 

1. Howdo experts assess the content validity of the Elementary School Teachers’ Environmental Pedagogi- 
cal Content Knowledge questionnaire in the Vietnamese context? 

2. Can the factor structure, as measured by the Elementary School Teachers’ Environmental Pedagogical 
Content Knowledge questionnaire, be confirmed in the Vietnamese context? 

3. To what extent do the factors in the Elementary School Teachers’ Environmental Pedagogical Content 
Knowledge questionnaire demonstrate reliability in the Vietnamese context? 


Research Methodology 
General Background 


The research was conducted over a three-month period from August to October 2023. A quantitative research 
approach was chosen to provide a systematic means of evaluating the questionnaire of teachers’ professional 
knowledge in environmental education. The study utilized a theoretical framework grounded in Pedagogical 
Content Knowledge (PCK), which highlights the integration of subject matter knowledge and pedagogical strat- 
egies necessary for effective teaching. This framework is essential in the context of environmental education, 
where teachers must not only understand environmental concepts but also know how to convey these concepts 
to students in an engaging and meaningful way. 
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Participants 


Drawing on the findings of prior research, content validity assessments are commonly conducted with the 
involvement of seven or more experts (Parsian & Dunning, 2009). However, Yusof (2019) proposed that a panel of 
five to ten experts for content validity assessment is generally sufficient, and the inclusion of more than ten experts 
might prove unnecessary. Considering these considerations, eight professionals affiliated with universities were 
invited to participate. Each of these experts had specialized knowledge in the fields of educational measurement 
(2 experts), elementary education (3 experts), and environmental education (3 experts). 

A convenience sample of one hundred elementary school teachers in the southern region of Vietnam was 
contacted and invited to participate in the study. There were 86 participants who responded, accounting for 
86.00% of the total number of invitees. The participant group was composed of 61 females (70.93%) and 25 males 
(29.07%), aligning with the gender distribution observed in Vietnamese elementary education sources (Nguyen, 
2020). Table 1 illustrates a comprehensive summary of the participants. 


Table 1 
The Detailed Information of the Participants 


Experts Elementary school teachers 
Gender 
n % n % 
Female 4 50.00 61 70.93 
Male 4 50.00 25 29.07 
Total 8 100.00 86 100.00 


Instrument and Procedures 


In this study, the“Elementary School Teachers’ Environmental Pedagogical Content Knowledge” questionnaire 
(Lee et al., 2018) was employed with the permission of the authors. This questionnaire is designed to assess seven 
latent constructs related to elementary school teachers’ professional knowledge in teaching environmental issues. It 
comprises a total of 39 items distributed across the following categories: 5 items for Content Knowledge, 5 items for 
General Pedagogical Knowledge, 4 items for Curriculum Knowledge, 5 items for Learners and Their Characteristics 
Knowledge, 6 items for Educational Contexts Knowledge, 8 items for Educational Ends, Purposes, and Values Knowl- 
edge, and 6 items for Assessment Knowledge. Participants were asked to express their degree of agreement with 
these statements using a 5-point Likert scale, which ranged from “strongly disagree” to “strongly agree” (see Table 2). 


Table 2 
Information of the Elementary School Teachers’ Environmental Pedagogical Content Knowledge Questionnaire 


Dimension Sample item Number ot 
Items 
CK know the inquiry process that a learner should have in environmental education. 1-5 
cmK understand the government-proposed environmental education standards for different grade levels of 6-9 
schools. 
LCK understand a lot of obstacles that students face when trying to adopt pro-environmental behavior. 10-14 
understand strategies that can be used in the classroom to make environmental concepts more acces- 
GPK : 15-19 
sible to students. 
understand the need to carry out environmental education in connection with social environmental 
ECK bee tae : ; 20-25 
education institutions (e.g., NGOs, local government, and business enterprises). 
PVK understand that a goal which considers the students’ long-term behavioral changes is necessary for 26-33 
environmental education. 
AK know how to assess affective domains such as attitude and belief towards the environment. 34-39 


Note. CK - Content Knowledge, GPK - General Pedagogical Knowledge, CmK - Curriculum Knowledge, LCK - Learners and Their 
Characteristics Knowledge, ECK - Educational Contexts Knowledge, PVK - Educational Ends, Purposes, and Values Knowledge, 
AK - Assessment Knowledge 
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The original questionnaire was written in English. Therefore, the authors of this study conducted the transla- 
tion process from an English questionnaire into Vietnamese to facilitate data collection. An English lecturer at the 
English Department, University of Education in Vietnam then back-translated this version into English to ensure 
its accuracy. After a thorough validation of the translation’s accuracy and linguistic equivalence, the Vietnamese 
version of the questionnaire underwent a pilot test with a committee of experts and elementary school teachers 
to assess the validity and reliability of the questionnaire items. 

In addition, a committee of experts utilized an additional evaluation tool known as the content validity form. 
This form enabled experts to assess the relevance and clarity of each item in relation to the study's objectives. This 
assessment was executed on a rating scale ranging from 1 (indicating “not relevant” or “not clear”) to 4 (indicating 
“highly relevant” or“highly clear”). Moreover, there was a section for experts to provide suggestions and comments 
on each item, further enhancing the comprehensiveness of their evaluation. 


Data Collection 


In the first round, the data collection process commenced in early August 2023 with the dispatch of formal 
invitations to eight university experts. These invitations allowed the experts to choose between an in-person or 
virtual approach. Three of the eight experts chose the in-person method, while the other six chose the virtual option. 

To facilitate the in-person meeting, a three-hour expert panel meeting was scheduled. During this session, 
the researcher guided the content validation process. On the other hand, experts were given an online content 
validation form as part of the virtual approach. To help with the content validation process, specific instructions 
were given in this context. 

In the second round, email invitations were extended to a sample of 100 elementary school teachers in 
September 2023. They were gently reminded of the invitation two weeks later. This strategy was implemented to 
encourage their participation in the study. By October 2023, this study collected responses from 86 teachers, who 
had willingly taken part in our research. To protect the privacy of participants, their personal information would 
be handled with confidentiality and used only for research purposes. 


Data Analysis 


To address the first question, the item-level content validity index (I-CVI) for every questionnaire item was 
calculated. The I-CVI values for each item were then averaged to calculate a scale-level content validity index based 
on the average method (S-CVI/Ave). In this context, and with the participation of eight experts, the minimally ac- 
ceptable scores for I-CVI and S-CVI/Ave were established as .78 and .83, respectively (Polit et al., 2007). 

To resolve the second question, data analysis was carried out using SPSS Statistics version 29. Extraction 
using Principal Component Analysis and rotation using Varimax were two of the data analysis methods used in 
exploratory factor analysis (EFA). These analytic procedures were instrumental in examining the construct validity 
of the research instrument. 

To answer the third question, this study continued to utilize SPSS Statistics version 29 to calculate Cronbach's 
Alpha coefficient. The following reliability levels are established by the interpretation of Cronbach's Alpha coef- 
ficient: invalid (.00 to .53), low (.54), moderate (.60 to .65), good (.66 to .71), excellent (.72 to .99), and perfect (1.00) 
(Creswell, 2010). 


Research Results 
Content Validity 


In alignment with prior research, items with an I-CVI value exceeding .79 were considered appropriate 
and retained. Moreover, items scoring within the range of .70 to .79 required modifications, guided by the recom- 
mendations provided by the panel of experts. Additionally, items scoring below .70 were removed as advised by 
Rodrigues et al. (2017). Results from the evaluation performed by eight specialists should not have an S-CVI value 
lower than .83, as per the standards laid out in the literature (Polit & Beck, 2006; Polit et al., 2007). Table 3 displays 
the content validity of the instrument in this study. 
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Table 3 
The Content Validity Index of the Instrument with Experts (N = 8) 


Factor Item Expert Agreement* I-CVI S-CVI/Ave Interpretation 
CK1 6/8 15 Need to revise 
CkK2 718 88 Accepted 
CK Ck3 8/8 1.00 88 Accepted 
CkK4 6/8 15 Need to revise 
CK5 8/8 1.00 Accepted 
CmK1 8/8 1.00 Accepted 
mak CmK2 8/8 1.00 a Accepted 
Cmk3 6/8 15 Need to revise 
CmkK4 8/8 1.00 Accepted 
LCK1 6/8 15 Need to revise 
LCK2 8/8 1.00 Accepted 
LCK LCK3 7/8 88 .90 Accepted 
LCK4 7/18 88 Accepted 
LCK5 8/8 1.00 Accepted 
GPK1 8/8 1.00 Accepted 
GPK2 8/8 1.00 Accepted 
GPK GPK3 6/8 15 95 Need to revise 
GPK4 8/8 1.00 Accepted 
GPK5 8/8 1.00 Accepted 
ECK1 6/8 15 Need to revise 
ECK2 7/8 88 Accepted 
ERK ECK3 6/8 15 i Need to revise 
ECK4 7/8 88 Accepted 
ECK5 8/8 1.00 Accepted 
ECK6 8/8 1.00 Accepted 
PVK1 7/8 88 Accepted 
PVK2 8/8 1.00 Accepted 
PVK3 8/8 1.00 Accepted 
oe PVK4 8/8 1.00 es Accepted 
PVK5 7/8 88 Accepted 
PVK6 6/8 15 Need to revise 
PVK7 6/8 15 Need to revise 
PVK8 6/8 15 Need to revise 
AK1 8/8 1.00 Accepted 
AK2 8/8 1.00 Accepted 
a AK3 8/8 1.00 i Accepted 
AK4 6/8 is) Need to revise 
AK5 5/8 63 Need to remove 
AK6 5/8 63 Need to remove 
Total .90 


Note. * The number of experts provides a score of 3 or 4 divided by the total number of experts 
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Table 3 presents the outcomes of the content assessment conducted by the experts on each item within 
the “Elementary School Teachers’ Environmental Pedagogical Content Knowledge” questionnaire. The total scale- 
content validity index stands at .89, indicating the overall appropriateness of the entire questionnaire. Out of the 
39 items examined, 26 items demonstrated acceptability and suitability, with I-CVI scores ranging from .88 to 1.00. 

Moreover, considerable discussion centered on the clarity and comprehensibility of certain questions. Items 
about “Content knowledge” (CK1, CK4), “Curriculum knowledge” (Cmk3), “Learners and their characteristics knowl- 
edge” (LCK1), “General pedagogical knowledge” (GPK3), “Educational contexts knowledge” (ECK1, ECK3),“Educational 
ends, purposes, and values knowledge” (PVK6, PVK7, PVK8), and “Assessment knowledge” (AK4) obtained a CVI score 
of .75. These specific items necessitated revisions, as advised by the expert panels, to align more effectively with 
the Vietnamese context. For example, the term “preconceptions” (original item LCK1) proved to be challenging for 
comprehension. Consequently, this item was rephrased as “prior knowledge” to enhance clarity and understand- 
ing. Also, in alignment with the Vietnamese Educational Curriculum, which outlines three general competencies, 
the ability to learn independently, to communicate and collaborate with others effectively, and to solve problems 
creatively (MOET, 2018), revisions were made to the original item PVK8. 

Nonetheless, two items (AK5 and AK6) were removed from the instrument because they were inconsistent 
with the assessment context in Vietnam. This modification was necessitated by the fact that environmental educa- 
tion is not presented as a single program in the Vietnamese Education Curriculum but rather is incorporated with 
other subjects. In addition, during the evaluation process, the panel of experts suggested reordering the item 
sequence within specific factors, such as “Content knowledge” and “Educational contexts knowledge’, to enhance 
the logical structure of the questionnaire. Finally, 26 items were accepted, 11 items were revised, and two items 
were removed for the new version of the instrument, called “Professional Knowledge for Teaching Environmental 
Issues” (PK-TEl) with 37 items. 


Construct Validity 


In order to determine whether the PK-TEI questionnaire was construct valid, an EFA was carried out. The com- 
monality values of all items in the questionnaire are above .72, hence there were no items to be removed from 
further steps of factor analysis (Costello & Osborne, 2005). After checking with elementary school teachers, the 
factor loading and item description for each of the 37 items included in the EFA is presented in Table 4. 


Table 4 
Exploratory Factor Analysis Results of PK-TE! Questionnaire in the Vietnamese Context (n = 86) 


Factor Item 1 2 3 4 5 6 7 Percentage of 
variance 


CK CK1 82 
CK3 83 
CkK4 16 
CmK CmK1 16 
CmkK2 93 7.78 
CmK4 62 
LCK LCK1 92 
LCK2 .90 
LCK4 95 
LCKS 89 
GPK GPK1 1 
GPK3 88 
GPK4 87 
GPK5 81 
ECK ECK3 94 
ECKS .90 10.84 
ECK6 94 


9.59 


13.69 


12.25 
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Percentage of 


Factor Item 1 2 3 4 5 6 7 ‘ 
variance 
PVK PVK1 88 
PVK2 87 
PVK3 88 
18.83 
PVK4 a 
PVK7 87 
PVK8 19 
AK AK1 83 
AK2 78 
11.91 
AK3 72 
AK4 88 
Total 84.89 


In Table 4, the results of the EFA based on the Principal Component Analysis extraction method and Varimax 
rotation method extracted seven factors of the PK-TEl questionnaire with eigenvalues greater than 1. In addition, 
some items were excluded because of small factor loadings (less than +.30) and cross-loading, certain items were 
deleted. At last, this study kept 27 items from the PK-TEI survey; these items are highly consistent with the authors’ 
and experts’ predicted domain recommendations. 

Particularly, the first construct, “Educational Ends, Purposes, and Value Knowledge’, explained 18.83% of the 
variance and included 6 items with standardized factor loadings between .71 and .88. The second construct, “Learn- 
ers and Their Characteristics Knowledge’, explained 13.69% of the variance and included 4 items with standardized 
factor loadings ranging from .89 to .95. The third construct, “General Pedagogical Knowledge’, explained 12.25% 
of the variance and included 4 items with standardized factor loadings between .71 and .88. The fourth construct, 
“Assessment Knowledge’, explained 11.91% of the variance and included 4 items with standardized factor loadings 
between .72 and .88. The fifth construct, “Educational Contexts Knowledge’, explained 10.84% of the variance and 
included 3 items with standardized factor loadings ranging from .90 to .94. The sixth construct, “Content Knowl- 
edge’, explained 9.59% of the variance and included 3 items with standardized factor loadings between .76 and 
.83. The seventh construct, “Curriculum Knowledge’, explained 7.78% of the variance and included 3 items with 
standardized factor loadings between .62 and .93. 


Internal Reliability 


The PK-TEI questionnaire’s reliability was tested using internal consistency reliability on a sample of 27 items. 
Table 5 displays the domain-specific results based on the seven factors. 


Table 5 
Reliability of the PK-TEI Questionnaire (n = 86) 


Factor Domain Number of items Cronbach’s a 
CK CK1, CK3, CK4 3 items 87 
CmK CmK1, CmK2, Cmk4 3 items 18 
LCK LCK1, LCK2, LCK4, LCK5 4 items 96 
GPK GPK1, GPK3, GPK4, GPK5 4 items 89 
ECK CK3, ECK5, ECK6 3 items 95 
PVK PVK1, PVK2, PVK3, PVK4, PVK7, PVK8 6 items 95 
AK AK1, AK2, AK3, AK4 4 items 92 
Total 27 items 86 
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As shown in Table 5, the PK-TEI questionnaire had 27 items and a Cronbach's Alpha value of a = .86. This value 
shows that the overall instrument has a high degree of internal reliability (Creswell, 2010). In detail, four subscales 
got excellent levels of internal reliability. The“Learners and their characteristics knowledge” subscale consisted of 4 
items (a = .96), the “Educational contexts knowledge” subscale consisted of 3 items (a = .95), the “Educational ends, 
purposes and values knowledge” subscale consisted of 6 items (a = .95), and the “Assessment knowledge” subscale 
consisted of 4 items (a = .92). Moreover, the three remaining subscales got good levels of internal reliability. For 
example, the “Content knowledge” subscale consisted of 3 items (a = .87), the “Curriculum knowledge” subscale 
consisted of 3 items (a = .78), and the “General pedagogical knowledge” subscale consisted of 4 items (a = .89). 

No items were removed from the reliability analysis based on these results. This led to the validation of the 
final version of the PK-TEI questionnaire, which comprises 27 items and covers seven primary factors: (1) content 
knowledge with three items; (2) curriculum knowledge with three items; (3) learners and their characteristics 
knowledge with four items; (4) general pedagogical knowledge with four items; (5) educational contexts knowl- 
edge with three items; (6) educational ends, purposes and values knowledge with six items; and (7) assessment 
knowledge with four items. 


Discussion 


To date, there has been a notable absence of studies in Vietnam that employ fully validated questionnaires to 
assess professional knowledge in environmental education (Thao et al., 2022). In response to this research gap, the 
main aim of this study was to adapt and validate an instrument that enables the evaluation of elementary school 
teachers’ professional knowledge in teaching environmental issues within the Vietnamese educational context. 
This process involved not only validating the existing questionnaire but also making necessary modifications to 
ensure its relevance and effectiveness for the specific educational and environmental context of Vietnam. 

This study has provided substantial evidence supporting the content validity of the questionnaire, originally 
developed by Lee et al. (2018) within the context of South Korean education. In their earlier work, the initial vali- 
dation of the questionnaire was conducted by a panel of ten experts, all operating within the framework of the 
Korean educational system, which led to the confirmation of 39 items. However, our study encountered certain 
challenges during the evaluation process, primarily in terms of modifying items that would offer clear and com- 
prehensive information to our respondents. The expert panel involved in this study raised concerns about the 
clarity of specific items. To enhance the quality of the questionnaire’s content, it needs to consider the educational 
context, particularly within the framework of the Educational Curriculum reform in Vietnam (MOET, 2018). As a 
result, we faced the demanding decision to eliminate two items and modify eleven others based on the feedback 
provided by the experts. These adjustments underscore the notable distinctions between the educational contexts 
of Vietnam and Korea. 

Regarding construct validity, an EFA was performed, and seven factors were identified: “content knowledge’, 


wu Ua 


“general pedagogical knowledge’, “curriculum knowledge’, “learners and their characteristics knowledge’, “edu- 
cational contexts knowledge”, “educational ends, purposes, and values knowledge’, and “assessment knowledge”. 
These elements are highly consistent with the original questionnaire’s predicted domains (Lee et al., 2018). How- 
ever, it is essential to note a difference between our study and previous ones: the questionnaire was reduced to 
27 items following the EFA. With a Cronbach's Alpha coefficient of .86, the questionnaire also demonstrated a high 
degree of reliability. This indicates that the instrument's items are internally consistent and reliably measure the 
intended constructs. 

In comparison with international studies, the findings of this study align well with the general trends observed 
in the validation of educational assessment tools. For instance, a study by Hill et al. (2008), the development and 
validation of a measure for teacher knowledge in the United States similarly faced the necessity of refining items 
to improve clarity and contextual relevance. Moreover, the reduction of items after EFA is a common step in the 
validation process, as evidenced by research conducted by Howard (2015), which also involved modifications to 
ensure the relevance and reliability of the instrument across different educational contexts. 

Further, the adaptation and validation process outlined in this study mirrors similar efforts in other cultural and 
educational settings. For example, a study by Lee et al. (2022) on the validation of an instrument to assess students’ 
perceptions of flipped classroom experience in Monash University Malaysia School of Pharmacy also highlighted 
the importance of contextual adaptations to align the instrument with local educational practices. This comparison 
underlines the global relevance of our approach and the necessity of cultural and contextual considerations in the 
validation of educational assessment tools. 
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It is important to note that while this study did make some noteworthy contributions, it did have several limita- 
tions. A primary limitation is the relatively small sample size of elementary school teachers (n = 86). Because of this 
limited sample size, this study was unable to perform Confirmatory Factor Analysis (CFA) and Structural Equation 
Modeling (SEM). CFA and SEM require a large sample size to ensure reliable results as suggested by Byrne (2006) 
and Hair et al. (2019). The current sample size was sufficient for exploratory factor analysis (EFA) but does not meet 
the recommended threshold for conducting CFA and SEM. Therefore, expanding the sample size would enhance 
the generalizability of findings concerning elementary school teachers’ professional knowledge in environmental 
education. Furthermore, it’s essential to recognize the potential selection bias due to the purposeful selection of 
experts and the convenient sampling of elementary school teachers. This bias may have contributed to the high 
coefficients of validity and reliability observed in the questionnaire. Therefore, further studies should employ dif- 
ferent approaches to provide additional validation for the instrument, such as CFA and SEM to confirm the factor 
structure and assess the model's fit. 


Conclusions and Implications 


This study effectively validated the instrument of Lee et al. (2018) in the Vietnamese context, thereby address- 
ing the critical need for an instrument to assess elementary school teachers’ professional knowledge of teaching 
environmental issues. Content validity, construct validity, and internal reliability were all part of the validation 
procedure that aimed to guarantee the reliability and validity of the questionnaire. 

The revised questionnaire, which consists of 27 items organized into seven constructs, provides a compre- 
hensive and well-structured instrument for assessing the professional knowledge of Vietnamese elementary 
school teachers. This instrument is useful for guiding teacher training programs and curriculum development to 
improve the successful implementation of environmental education in elementary schools. The seven constructs 
identified—content knowledge, general pedagogical knowledge, curriculum knowledge, learners and their char- 
acteristics knowledge, educational contexts knowledge, educational ends, purposes and values knowledge, and 
assessment knowledge—cover critical areas essential for effective environmental education. 

Although the study’s primary objective was to check the questionnaire in the Vietnamese context, its impli- 
cations extend beyond Vietnam. Other countries seeking to assess and improve the environmental professional 
knowledge of elementary school teachers can adopt and adapt this instrument to fit their educational contexts. 
The need for a culturally and contextually relevant tool for evaluating teachers’ knowledge in environmental educa- 
tion is universal, given the global challenges of climate change, resource depletion, and ecological deterioration. 

Recognizing the critical role that elementary school teachers play in developing environmental consciousness 
in their students, the results stress the importance of investing in their professional development as environmental 
education facilitators. By equipping teachers with the necessary knowledge and skills, they can effectively integrate 
environmental education into their teaching practices, thereby fostering a new generation of environmentally 
conscious citizens. This research contributes to ongoing global efforts to strengthen environmental education in 
elementary schools and supports the broader objective of promoting sustainable development through education. 

Moreover, the validated instrument supports educational policymakers and curriculum developers in iden- 
tifying specific areas where teachers may need additional training or resources. This targeted approach can lead 
to more effective and meaningful professional development programs, ultimately enhancing the quality of envi- 
ronmental education delivered to students. 

This study not only provides a validated tool for assessing elementary school teachers’ professional knowledge 
in Vietnam but also offers a model for similar efforts in other countries. The findings underscore the necessity of 
culturally and contextually relevant instruments in educational research and highlight the global importance of 
empowering teachers to address environmental issues through education. 
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Abstract. Systems thinking (ST) is 
an essential skill for understanding 
complex issues, making predictions 

and informed decisions. This research 
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complex health contexts using the ‘One 
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interdependence relationship between 
animal, human and ecosystemic health 

(including plants). Eighteen upper 
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capacity to apply ST to explain the causes 
of the COVID-19 pandemic, to anticipate 
future pandemics and to propose actions 
for preventing them. Individual written 
tasks were examined using content analysis 
methods. Four dimensions of ST were 
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based on the literature and in interaction 
with data. Most participants articulated 
various aspects of ST in their responses, but 
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had difficulties both in anticipating future 
pandemics and in proposing actions 

to prevent them. After the completion 

of the activities, most students showed 

an improved understanding of the OH 
notion. They identified the components, 
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causal interrelationships, such as those 
involved in zoonotic diseases. The findings 
point to the potential of the OH approach in 
Biology education, as it promotes students’ 
understanding of complex health issues 
from a systemic view. 
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THE APPLICATION OF THE ONE 
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Introduction 


Many health issues are “systemic problems” that affect human, envi- 
ronment and animal health; thus, their understanding might benefit from 
the integration of the ‘One Health’ (OH) approach in Biology education. The 
United Nations Environment Assembly that took place in 2022 recognized 
the risk of future pandemics and other health concerns. This risk is fostered 
if humans do not revise their patterns of interaction with nature by adopting 
a holistic approach such as OH (Food and Agriculture Organization of the 
United Nations [FAO] et al., 2019; FAO et al., 2022). OH takes into account the 
interrelationships between the health of animals, ecosystems and humans. 

Thus, Biology education, in line with all disciplines forming school cur- 
ricula, should promote students’ development of systems thinking (ST) as 
part of their health literacy to take actions that benefit them, the community 
and the planet. The World Health Organization (WHO, 2021) provided the 
following definition of health literacy: 


“Health literacy represents the personal knowledge and competencies 
that accumulate through daily activities, social interactions and across 
generations. Personal knowledge and competencies are mediated by the 
organizational structures and availability of resources that enable people 
to access, understand, appraise, and use information and services in ways 
that promote and maintain good health and well-being for themselves 
and those around them’ (p.6). 


Even though the notion of OH has been around for some time in the 
scientific community and political institutions, it is only recently gaining 
momentum in the public eye regarding the teaching of Biology. In Biology 
education, there are few studies that address the OH approach (Uskola & 
Puig, 2022) particularly in teaching and learning complex health issues, even 
though the COVID-19 pandemic has highlighted the importance of promot- 
ing the application of OH to understand these problems from an integrated 
viewpoint (Byrne et al., 2022; Ronner et al., 2023a, 2023b). 
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The current Biology curriculum for secondary education in Spain, aligned with the Sustainable Develop- 
ment Goals (SDGs) proposed by the United Nations (United Nations, 2015), supports incorporating OH in Biology 
education. However, teachers are not being provided with tools or examples of learning situations that can help 
to engage students in its application to real-life contexts. This study advocates that ST is necessary to understand 
the OH notion, as ST has helped to address the linear and reductionist approaches that prevail in health problems 
(Atun, 2012) and comprehend the factors that interact and influence the complexity of these issues. 


The Integration of the ‘One Health’ (OH) Approach in Health Literacy 


The COVID-19 pandemic evinced that we live in a globalised world, where human public health problems 
are no longer restricted to specific geographical areas and that our health is closely related to animal and envi- 
ronmental health. This global view is required to improve students’ health literacy. From today’s health literacy 
perspective, social, economic, and environmental issues become as important as individual factors. Nutbeam 
(2000) distinguished between functional health literacy, interactive health literacy, and critical health literacy. 
Functional health literacy consists of providing information about risks and about health systems; interactive 
health literacy focuses on generating motivation and self-confidence; critical health literacy seeks social and 
political action in addition to individual action. The need for actions is particularly relevant in the context of 
health education and was taken into account by the Organization for Economic Co-operation Development 
(OECD) in the definition of the Learning Compass 2030 (OECD, 2018) that aims to help students to navigate to- 
wards future well-being. As Levrini et al. (2020) suggested, agency and awareness of the need to take responsible 
actions emerge through anticipatory and reflective processes, and perception of the future involves personal 
engagement (Levrini et al., 2015). 

The definition given by WHO (2021) mentioned above, is coherent with the critical view, but does not include 
references to animal nor environmental health, dimensions included in the OH notion, in spite of the fact that 
OH was introduced at the beginning of this century. OH was established by the OH High-Level Expert Panel, 
and it recognizes that the health of humans, domestic and wild animals, and the wider environment (including 
plants and ecosystems) are closely linked and interdependent (FAO et al., 2019). Hence, OH is a wide-ranging 
concept that aims to integrate different disciplines, and analyses health at different levels: that of the individual, 
the population, and the ecosystem (Lerner & Berg, 2015). Following this approach, taking into consideration the 
interactions between animals-humans-ecosystems when assessing a health-related issue from a multidisciplinary 
perspective, will help improve health at the above-mentioned levels. 

In the case of health literacy, if the OH approach (FAO et al., 2019) is considered, in addition to individual 
actions, social and political ones are also needed to protect the health of both ecosystems and animals, as well 
as public health. It can thus be said that critical health literacy (Nutbeam, 2000) becomes more relevant when 
taking an OH approach, and this should be addressed by education stakeholders. 

Despite all governmental agency support (WHO, OIE, FAO, etc.), OH still needs to be incorporated into 
the educational system (Barret et al., 2010), and this requires that educational policies move towards this goal. 
However, educational institutions have not provided specific tools to promote this approach. An example of 
the shortages is that textbooks usually neglect zoonoses, as a Delphi study focused on contagious diseases 
that can lead to epidemics and pandemics pointed out (Kilstadius & Gericke, 2017). Informing students about 
zoonoses is coherent with the OH perspective, as understanding the emergence of zoonoses involves being able 
to identify the interdependent relationships between animal, ecological, and human health (FAO et al., 2019). 

A recent publication has shown an increasing awareness among students towards health issues due to the 
COVID-19 pandemic outbreak (R6nner et al., 2023a). Despite the increasing inclusion of health issues in science 
learning, students have shown some limitations related to essential concepts of health education, such as the 
concept of ‘infectious disease’ the differences between virus and bacteria, or the relevance, efficacy, and func- 
tions of vaccines (Abramczyk et al., 2023; Calavia et al., 2022; Ronner et al., 2023a, 2023b). 

The OH approach involves the development of ST to address complex health issues, since addressing them 
requires identifying the elements related to humans, animals and the environment, and analyzing how they 
interact and result in health-related phenomena such as pandemics. 
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System Thinking (ST) for Addressing Complex Health Issues 


The development of ST has been a focus in science education for a long time (i.e. Ben-Zvi-Assaraf & Orion, 
2005; Hmelo-Silver & Pfeffer, 2004) and is achieving a growing presence in the science classroom (Ben-Zvi-Assaraf 
& Knippels, 2022). A recent work published by Momsen et al. (2022) proposes the use of ST to unify the learning 
of complex phenomena in Biology, synthesizing and extending the contributions of previous works (Ben-Zvi- 
Assaraf & Orion, 2005, 2010; Momsen et al., 2022; Sommer & Lucken, 2010). Research has shown the benefits of 
incorporating ST into science education compared to other teaching methodologies based on memorizing and 
reductionist approaches (i.e. Jacobson & Wilensky, 2006; York et al., 2019). As Hannon and Ruth (2000) stated, 
current science education still focuses on events rather than on processes over time, on parts rather than on 
systems and on isolated processes rather than on systemic relationships. 

According to Ben-Zvi-Assaraf and Orion (2010), ST allows for the tackling of complex issues in a holistic 
way, identifying the factors involved, and the interdependent relationship between them. Ben-Zvi-Assaraf 
and Orion (2005) proposed that ST requires the development of a set of eight ST skills such as the following: 
identifying system components and processes; identifying the relationships between the components of the 
system; identifying the dynamic relationships within the system; developing Temporal thinking (retrospectively 
and prospectively). 

From a Temporal thinking standpoint, a system is the result of several interactions that took place in the 
past, and the future of the system will be the result of the interactions that are being developed in the present. 
As Levrini et al. (2021) claimed, decisions and actions become directionless if they are not based on any image 
of the past, nor aimed at any stable future horizon. Complex health issues such as pandemics need to be ad- 
dressed by connecting past and present events to make decisions based on these experiences to face potential 
future scenarios. Bielik et al. (2023) used concept maps elaborated by students to analyze their ST abilities on 
the COVID-19 pandemic, focusing on whether they referred only to biological aspects, or also to social aspects. 
They found that students reflected little in the dynamic sense (related to Temporal thinking). 

Mehren et al/s (2018) two-dimensional model highlights this Temporal thinking, as the dimensions proposed 
were: 1) Retrospective dimension (ST-OrgBeh) that includes issues related to the understanding of the system 
and its functioning (components and relationships), and that reflects knowledge of the system; 2) Prospective 
dimension (ST-Action) that refers to the ability to make forecasts and propose appropriate actions, which is 
related to the capacity to anticipate future events. 

The topics used for the analysis of ST in Biology and Geology education were related to Earth, as an example 
of acomplex, dynamic and changing system; this included topics such as the rock cycle (Ben-Zvi-Assaraf & Orion, 
2005) and other geologic issues regarding climate change (Kali et al., 2003), ecosystem dynamics (Hmelo-Silver 
et al., 2017; Li & Li, 2023: Mambrey et al., 2022), the water cycle (Ben-Zvi-Assaraf & Orion, 2010) and the systemic 
nature of the human body (Tripto et al., 2013). 

Tackling health-related issues from an ST and OH approach in science education is a nascent line of research 
(Puig & Uskola, 2021, Uskola & Puig, 2022, 2023). Uskola and Puig (2021) and Uskola and Puig (2022) performed 
diagnostic studies to explore pre-service teachers’ ST and OH notion. The activities designed by Uskola and Puig 
(2023) for participants to develop ST were also conducted with pre-service teachers. Nevertheless, studies in 
secondary education, in which curricula ST and OH are gaining presence, are limited. 


Research Aim and Questions 


This work sought to make a contribution to this emergent field of research by exploring how secondary 
school students mobilize ST to address complex health issues, such as the COVID-19 pandemic, based on the OH 
approach. To achieve this aim, the following research questions (RQ) were proposed: 

(RQ1) To what extent do secondary school students apply ST when identifying potential causes of the 
COVID-19 pandemic? 

(RQ2) What are students’ perceptions regarding the anticipation of future pandemics? And in what way are 
these perceptions linked to their ability to identify potential causes of future epidemics and to their 
recommendation of actions for preventing them? 

(RQ3) How do students apply ST to define the OH notion after their engagement in activities that require 
its application? 
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General Background 


The research consisted of a qualitative study that involved the collection and interpretation of data regard- 
ing the understanding of complex health issues, particularly the COVID-19 pandemic from a holistic view (Gibbs, 
2012), as is the OH approach. 

A case study, understood as an in-depth exploration from multiple perspectives of a particular case (Simons, 
2011) was developed. Participants were all secondary school students (14-15 years old) in a state school classroom. 
They were engaged in a set of activities that were integrated into the current Biology curriculum with the objective 
of developing an understanding of the COVID-19 pandemic and developing ST skills in a complex health context 
using the OH approach, in March-April 2022. 


Participants 


The participants were from the same, single class of students (10 female, 8 male) chosen for convenience as the 
first author was one of the teachers at the time of implementing the activities. They were of amedium/medium-high 
socio-economic status and lived in urban or rural areas close to the city in which the educational centre was located. 
According to the Biology teacher, the group dynamics were good and showed interest in learning Biology topics. 

All participants were properly informed by the researchers before starting the implementation. Their parents/ 
legal guardians were provided with an informative document and the informed consent document was signed by 
the parents/legal guardians of each participant, according to the ethical law existing at the time of implementation. 


Design and Implementation of a Teaching Sequence 


A teaching sequence was designed by author 1 in collaboration with the other two co-authors and discussed 
with the Biology teacher. The sequence aimed to engage students in the construction of knowledge and the de- 
velopment of ST skills using the OH approach in the context of the COVID-19 pandemic. Furthermore, the activities 
aimed to make students aware of the role of human beings in the emergence of infectious diseases, and in the 
actions we can take to minimize the risk of the emergence of new diseases (agency). 

The sequence was integrated in Biology teaching as part of the complex health issues learning. It should be 
considered that this is the first time that the participants learned about health topics from an OH approach. Table 
1 shows the activities, their distribution in lessons, the learning goals, and their use for addressing the RQs. 


Table 1 
Activities of the Teaching Sequence 


Activity/ Individual (I)/ F 
Lesson Task Group(G) Learning goals RQ addressed 
, T1.1 G To express key aspects that define the notion of health 
71.2 G To identify diverse infectious agents and to explain how to treat them 
T2.1 G To explain infectious diseases 
12.2 | To explain the main factors that influence the occurrence of the ROI, ROB 
COVID-19 pandemic and the relationships between them 
134 To apply ST to identify the main causal factors of the COVID-1 9 RQ1, RQ2, ROB 
: pandemic and the potential emergence of future pandemics 
13.2 G To identify other examples of diseases that could potentially become 
, pandemics 
74.1 G To reflect on the consequences of human activities on public health 
4 To critically analyze the information provided in diverse case studies 
14.2 G oo 
related to zoonotic diseases 
5 T5 | To define OH RQ3 
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Students worked individually and in small groups (3-5 students). In the beginning, the first three tasks, T1.1,T1.2 
andT2.1, were carried out in these groups (Figure 1) to facilitate the sharing of their ideas. T2.2 (“Draw up a scheme 
or diagram in which you indicate the factors that you think influenced the COVID-19 pandemic. They should also 
indicate the relationship between the factors, if known") required the individual production of a diagram or drawing. 


Figure 1 
Group of Students Working in T2.2 


T3.1 consisted of an individual questionnaire designed by the authors based on previous works on ST (Uskola 
& Puig, 2022, 2023). It was designed to encourage students to think about the potential causal factors of the origin 
of the COVID-19 pandemic and explain the likelihood of the emergence of new pandemics in the future. Table 2 
shows the questions and the aspects analyzed linked to the RQs. 


Table 2 
Questions in T3.1 Analysed for Addressing the RQs 


Questions RQ addressed Dimensions 

T3.1.4 What individual actions do you think can help prevent new pandemics, F 

such as COVID-19? he een 

T3.1.5 Do you think this COVID-19 pandemic was a one-off event or do you think RQ1 ST (Temporal thinking) 

more pandemics may occur in the future? Give your reasons. RQ2 Anticipation 

T3.1.6 What do you think are the cause(s) of the current COVID-19 pandemic? ped i Companetie) REI VONSENS, 
RQ3 Causes) 

T3.1.11 Do you think humans can anticipate the emergence of new pandemics? RQ2 Anticipation 


T4.1 was designed to help students establish relationships between human actions and the emergence of 
infectious diseases. T4.2 consisted of the analysis of diverse hypothetical cases of zoonosis designed by author 1, 
inspired by the resource developed by Michigan State University (2023). 


Data Collection and Analysis 


Data included the written reports produced by the students during the development of all activities. Addi- 
tionally, joint discussions during the activities were audio-recorded, as Figure 1 shows, and field notes were taken. 
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For RQ1, diverse sources of data were used: the diagrams elaborated by students individually in T2.2, as well as 
individually written responses to T3.1. The content of the diverse sources of data was analyzed in order to examine 
the ST shown by students. To do such an analysis, the characteristics of ST defined by Ben-Zvi-Assaraf and Orion 
(2005) and Mehren et al. (2018) were taken into account. Thus, four dimensions of ST were considered, and the 
levels were established in interaction with data for each ST aspect which are shown in Table 3: 

1) Components: this refers to the elements of the system (Li & Li, 2023). In this study, these components 
concern the three spheres (human, environmental, and animal health) of the OH notion (FAO et al., 
2019). 

2) Interactions: this corresponds to the identification of the relationships or connections between the 
components of the system (Ben-Zvi Assaraf & Orion, 2005), in this case, among the elements of the 
three OH spheres. 

3) Causal relationships: this refers to the cause-effect relationships (Mehren et al., 2018) that students estab- 
lished among the elements of the three OH spheres. As the Interactions aspect includes co-occurrence 
but does not necessarily involve causality, we also studied the identification of causal relationships as 
an essential dimension of ST. The distinction between interactions and cause-effect relationships has 
been made in previous works such as the one cited by Mehren et al. (2018) and more recently by Bielik 
et al. (2023) in the context of applying ST to the conceptualization of COVID-19. A distinction was made 
between these two levels: ‘simple structural relationships’ and ‘simple mechanistic relationships. 

4) Temporal thinking: as mentioned above, this refers to the ability to connect past and present experi- 
ences/events to the future (Ben-Zvi-Assaraf & Orion, 2005; Mehren et al., 2018). 


Table 3 
Dimensions and Levels for ST Analysis 
ST dimensions Levels 


Level 3. Three spheres (H-E-A) 


Level 2. Two spheres 


Components 

Level 1. One sphere (A, H or E) 

Level 0. No sphere 

Level 1. Identifying interactions among elements 
Interactions 


Level 0. Not identifying interactions among elements 


Level 1. Establishing cause-effect relationships 
Causal relationships 
Level 0. Not establishing cause-effect relationships 


Level 2. Connecting with past and present 


Temporal thinking Level 1. Connecting with past/present 


Level 0. Not connecting with past/present 


Concerning RQ2, the analysis involved three steps and the levels were established taking the data into ac- 
count. First, students’ perceptions about Anticipation (Table 4) were explored. Anticipation can be defined as the 
ability to predict future challenges based on past events, current evidence and knowledge (Deans, 2021). This is 
a fundamental skill of ST, and essential for fostering decision-making. The responses to question T3.1.11 (Table 2) 
were assigned to a particular level (Table 4). Level 0 was assigned to those students who said it was impossible to 
anticipate future pandemics. Level 2 was assigned to those who said it was possible, and vague responses were 
assigned to Level 1. 

Secondly, the actions proposed by students to question T3.1.4 concerning how to prevent future pandemics 
were examined. The responses were classified as mitigation actions (Level 1), need for information (Level 2) and 
actions directed to causes (Level 3) (Table 4). Finally, this research explored whether the students identified the 
potential causes of future pandemics when making predictions in question T3.1.5, considering that the ability to 
identify potential causes of a specific problem also shows a high degree of ST development. 
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Dimensions and Levels for Analysing Students‘ Perceptions on Pandemics 


Dimensions of students’ perceptions Levels 
Level 2. It is possible to anticipate future pandemics, and the answers are justified 
eT Level 1. It is possible to anticipate future pandemics, but vague and not justified 
Anticipation : 
answers were provided 
Level 0. It is impossible to anticipate future pandemics 
Level 3. Actions directed to causes 
Level 2. Need for information 
Actions 
Level 1. Mitigation actions 
Level 0. No actions proposed 


Concerning RQ3, the analysis applied the dimensions and levels shown in Table 3. These levels were used to 
examine the responses to T5 (OH context) and to compare them with the COVID-19 context (highest level in T2.2 
and T3.1). This analysis was performed to identify the evolution of the students’ conception of OH and, hence, the 
evolution of ST-related skills. 


Research Results 
Application of ST in the Identification of COVID-19 Pandemic Causes 


Table 5 shows the frequency of students’ (S) responses in each level regarding ST in each of the questions 
analyzed. 


Table 5 
Frequency of Students in Each ST Dimension and Level 


Activity/Task 
ST dimension Level 
72.2 73.1.6 T2.2/T3.1.6 73.1.5 
3 0 0 0 
2 10 7 15 
Components 
1 8 1 3 
0 0 0 0 
1 5 6 10 - 
Interactions 
0 13 12 8 - 
1 5 7 10 - 
Causal relationships 
0 13 1 8 - 
2 - - - 2 
Temporal thinking 1 - - - 12 
0 - - - 4 


Note. T2.2/T3.1.6 column shows the highest level reached. 


As is shown in Table 5, most students mentioned two of the three OH spheres when explaining the COVID-19 
pandemic (Components, Level 2). The animal sphere was the most cited after the Human sphere, while only one 
student referred to the Environment. None referred to all three spheres (Components, Level 3). Only three students 
mentioned two spheres in both questions. In the following example, S2 referred to Animal and Human spheres 
(Interactions, Level 1). 


$2 (T3.1.6): ‘Il think a virus that used to be in animals mutated and made people infected.’ 
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More than half of the students identified Interactions between the components at least in one of the questions, 
but most students did so only when responding to one of the two questions, and only one student did so in both 
responses. The response of S2 given above shows a relationship between the spheres but the following example 
of S10 does not (Interactions, Level 0), although $10 did refer to Animal and Human spheres: 


$10 (T3.1.6): ‘A mutation of a virus that affected animals. Then, a massive contagion due to misinformation in the early days.’ 


Similarly to the findings in Interactions, more than half of the students identified Causal relationships at least 
in one of the questions (Causal Relationships, Level 1), but the majority of students-did so only when responding 
to one of the two questions, and only one student did so in both responses. The response of S2 given above shows 
that S2 identified the origin of a zoonotic disease (Causal Relationships, Level 1). Besides a zoonotic origin, an ac- 
cidental or deliberate origin in a laboratory was the other type of cause identified, as the example of S16 shows: 


$16 (T3.1.6):‘l believe it was a virus created to decrease the world’s elderly population in an attempt to avoid a major 
economic crisis, which would explain the rapid creation of the vaccine and the absence of cases in the initial country 
(China): 


Figure 2 shows the diagram elaborated by $17 in response to T2.2. 


Figure 2 
Diagram made by $17 in T2.2 


Note. Translation to English: (1) arrives at; (2) ANIMAL (in China); (3) Virus mutates, that is, changes; (4) Animal arrives for example 
at a market; (5) Bad health habits; (6) markets with animals in poor condition; (7) one person eats an animal in poor condition; (8) Bat, 
pangolin; (9) a person contracts the mutated virus (SARS-CoV2); (10) presents new symptoms; (11) spreads to people close to him; (12) 
Lack of similar experiences; (13) The virus continues to spread; (14) Becomes a global pandemic; (15) Vaccine research begins; (16) They 
propose social restrictions (confinement and use of masks); (17) the advance of the virus is slowed down; (18) a vaccine finally appears; 
(19) the pandemic situation is improving PRESENT. 
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It can be observed in Figure 2 that $17 drew a diagram in which he progressively explained the potential 
causes of the pandemic by connecting (using a sequence of arrows) diverse events that are part of human and 
animal spheres. S17 referred to the causal relationship (Causal Relationships, Level 1) between animal and human 
health (‘one person eats an animal in poor condition; one person contracts the mutated virus SARS-CoV2’, (the person) 
spreads it to people close to him’. 

Regarding Temporal thinking, it is important to highlight that all students predicted that pandemics will oc- 
cur in the future, and the majority (12/18) referenced either past (7/18) or present (5/18) events to elaborate and 
justify their answer, as the following examples show: 


S4 (past): I think that this was not a one-off event and that more may occur in the future. | think this because there have 
been others in the past and there is no way of knowing whether there will be more in the future, although it is most likely. 


S7 (present): ‘I believe that more pandemics may occur in the future as new diseases and new viruses appear every week. 


S4 based the response on past events, whereas S7’s prediction was based on the present. Both were coded 
in Temporal Thinking Level 1. 


Students’ Perceptions About Anticipating Future Pandemics 


Table 6 shows the results of the analysis of students’ perception concerning the possibility of anticipating 
future pandemics, with the frequencies of each level and examples that illustrate them. 


Table 6 
Frequency of Students ’ Anticipation Responses 


Level Frequency Example 


$17: ‘Yes, | think that even if humans do not know exactly when and where a 
2 Possible to anticipate 4 new pandemic will start, what they can do is to prevent them, by increasing 
safety with more hygiene and so on’. 


$10: ‘I’m not sure but it's possible that somehow they can be foreseen, 


Nobles although not with much certainty, although I’m not sure’. 
0 Impossible to anticipate 8 $2: 1 think there is nothing we can do about it because anything can cause it’. 


Not answer 1 - 


The perception of most students (8/18) was that it is not possible to anticipate future pandemics (Level 0), 
whereas the minority (4/18) considered it possible to anticipate them (Level 2). ln the category of ‘Not clear’ (Level 
1), five responses were included. All of them showed doubts about the possibility of anticipating future pandemics, 
although they did not develop their responses by providing justification for their answers. 

Only 7/18 students were able to identify the potential causes of future pandemics. These included: muta- 
tions (3/18), emergence of new viruses and diseases (2/18), lack of health measures (1/18), and the way we act in 
the world (1/18). Regarding the actions to prevent future pandemics, most students (13/18) proposed immediate 
actions such as the ones applied to prevent the spread of COVID-19: sanitary and safety measures, for example, 
using masks and keeping your distance. Four students pointed to the need to be ‘well’informed to act responsibly 
against future pandemics. 

No student proposed actions to attack the causes of pandemics. The rest of the students (4/18) did not pro- 
pose any actions; indeed, three of them argued that future pandemics cannot be avoided since they are natural 
events. The analysis of the potential relationship between students’ perception of the possibility to anticipate future 
pandemics and the identification of causes and actions is displayed in Figure 3 (individual responses). 
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Figure 3 
Students’ Perceptions about the Capacity to Predict Future Pandemics and Students’ Identification of Causes of Pandemics 


and Actions to Prevent Them. 


Level of development regarding: cause identification, 
anticipation, and actions 


UM Sees | 
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As can be seen in Figure 3, all the students who identified causes for future pandemics perceived that it was 
not impossible to anticipate them. Only one student (S15) did not respond. Additionally, the four students who 
did not propose any action considered that we cannot anticipate future pandemics, nor did they identify potential 


causes of future pandemics. 


Students’ OH Notion 


Figure 4 shows the responses of 14 students to question T5. Four students did not answer. 
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Figure 4 
Frequency of Students ’ Responses in each Level of ST Dimensions in the OH Context 
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Note. a) Components, b) Interactions, c) Causal relationships. 


Figure 4a shows that 12/14 students mentioned the three spheres of the OH notion, as well as reaching the 
highest level for this aspect (Level 3), which no students did in the COVID-19 context initially (Table 5). The majority 
of students identified interactions between the spheres (Figure 4b) and reached Level 1. This dimension also showed 
better performance in the final context than in the initial one, as the comparison between Table 5 and Figure 4b 
shows. Indeed, they used words such as dependence (6), interrelationship (5), affect (3) and interconnection (1). 

Moreover, 10/14 gave examples where they established causal relationships for the occurrence of pandem- 
ics reaching Level 1 (Figure 4c). Similarly, in the case of causal relationships, the results were better in the final 
context than in the initial one, as the comparison between Table 5 and Figure 4c shows. The following example 
of $14 illustrates this: 


S14: ‘The concept of OH advocates that the health of animals, people, and ecosystems/environment are interrelated, so 
what happens in one will affect the others. 

For example, if we destroy the health of ecosystems and the environment and deforest a place, the animals that used to live 
there can go to other places and infect people with their diseases. Destroying the health of the environment would have an 
effect on the health of animals and humans. 


Figure 5 shows the level reached by each student in both contexts, the initial COVID-19 context, and the final 
OH context, with reference to ST dimensions. 
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Figure 5 
Comparison of the Degree of Students‘ ST Development between the Initial Context (COVID-19 Context) and the Final Context 


(One Health Context). 
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As can be seen in Figure 5, two students, S11 and S16, did not develop a notion of OH: $11 thought that it 
referred to only presenting one kind of human health, for instance, physical or mental, but not both, and S16 re- 
ferred to health being delicate and easy to lose. Regarding the rest of the students, S12 was the only one that was 
able to identify the three spheres but was not able to interrelate them. Comparing the OH context with the initial 
COVID-19 context, some noteworthy answers can be seen in Figure 5, such as those of $3 and S5 that only referred 
to human aspects initially and showed a complete OH understanding at the end of the sequence. 


Discussion 


This study aimed to explore how a group of secondary school students mobilized ST in order to address 
complex health issues such as the COVID-19 pandemic using the OH approach. Regarding RQ1, students identified 
two of the three components of the OH notion, mainly animal and human health. Furthermore, more than half 
identified interactions between the components, and showed simple causal thinking. For example, they attributed 
the pandemic to a single cause, either a zoonosis or its creation in a laboratory. Nevertheless, the results are better 
than those in a previous study (Puig & Uskola, 2021), in which pre-service teachers focused on factors related to 
the health measures implemented after the onset of the COVID-19 pandemic, mainly focusing on human health. 
In this case, participants showed awareness of zoonosis, which is fundamental for developing contagion and 
health literacy (Kilstadius & Gericke, 2017). At the time this study was implemented, the notion of zoonosis was 
more mediatic than the time of the previous one (Puig & Uskola, 2021). This may affect the results since students 
could be more familiar with zoonosis. 

The analysis of ST related to other areas of Biology and Geology, such as the human body (Snapir et al., 2017), 
ecology (Mambrey et al., 2020; Riess & Mischo, 2010) or geology (Ben-Zvi-Assaraf & Orion, 2005), has generally 
shown a higher degree of ST than that observed in this work: one example, regarding the most basic feature of 
ST, is the identification of system components. This may be explained by the complexity of considering so many 
factors (environmental, human and animal) that were present in the COVID-19 pandemic. 
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Students pointed to two alternative explanations regarding the potential causes of the COVID-19 pandemic: 
‘the virus was created or escaped from a laboratory’ and ‘the zoonotic origin of the infectious disease, which cor- 
respond to human and animal spheres, respectively. This is coherent with the information disseminated by the 
media and reflects students’ questioning on this issue, considering it to be an unsolved problem. ST aims to help 
students to understand and to explain the systemic nature of natural phenomena. However, this study showed 
that when controversy affects even the identification of system components, other ST dimensions (e.g., connecting 
elements of a system) are difficult to activate. These results also showed the need to specifically promote ST-related 
skills for health literacy in Biology education. 

Addressing RQ2, only four students believed that it was possible to anticipate future pandemics, although all 
18 students responded that they thought that future pandemics would indeed occur. Anticipating future pandemics 
is crucial for preventing them, and the ability to anticipate is part of future thinking (FT) skills (Levrini et al., 2021). 
Anticipating the future, imagining the future, was the dimension that the students participating in Levrini et al’s 
(2021) study were least able to develop. According to these authors, FT ‘refers to mid-long term thinking and the 
‘imagination dimension’ which are particularly difficult to develop in a short time’ (1p. 297). 

As the results reflect, this ability to anticipate is linked to the identification of causes, as all the students who 
identified potential causes of future pandemics believed that it was possible to anticipate them. Despite the limita- 
tion of the sample in this study, the relationship between the ability to anticipate future events and the ability to 
propose actions (agency), is shown by the results: the four students who did not propose any action considered 
that we cannot anticipate future pandemics and did not identify causes of future pandemics either. 

Moreover, the results show that in the cases in which students identified causes of future pandemics, a few 
referred to virus mutations, but did not consider the causes that they had pointed out in the case of COVID-19. 
For instance, no student referred to animal health problems as being possible causes of pandemics. This issue can 
also help to explain the type of actions that the students suggested for preventing pandemics; that is to say, no 
student proposed actions aimed at addressing the causes of pandemics (e.g. zoonoses). Indeed, they mostly limited 
themselves to proposing hygiene measures or the need for information. Students used their own experiences to 
propose direct actions such as the ones they carried out to prevent the spread of COVID-19 (e.g., washing their 
hands, using face masks, keeping their social distance, etc). This is also coherent with previous research (Uskola & 
Puig , 2022), and reflects that personal experiences play a predominant role, in contrast to indirect actions that 
involve policy measurements: students revealed some “functional” health literacy but a lack of “critical” health 
literacy (Nutbeam, 2000). This research therefore highlights the need to provide a wider, more complex view of 
health problems in order to foster student agency. 

RQ3 showed that the definitions of OH presented by the participants reflect an awareness of the relationships 
between the three spheres of this notion. They also highlight students’ capacity at the end of the activity sequence 
to apply this notion to other health-related examples, such as zoonoses and the occurrence of pandemics. This 
positioned them as expert learners in ST (Hmelo-Silver & Pfeffer, 2004). 

Initially, they were not able to transfer what they knew about the origin of the COVID-19 pandemic to future 
pandemics. This is shown by the fact that they related to some extent the causes of the COVID-19 pandemic to 
our relationship with animals (to a lesser extent with the environment). However, when it came to justifying the 
possible appearance of future pandemics, they referred to mutations, the appearance of new viruses, but hardly 
at all to the relationship of humans with animals and ecosystems, or to the consequences of human actions. This 
made it difficult for them to anticipate future pandemics and to propose actions to prevent them. None of the 
actions proposed revealed the application of the OH approach. 

In contrast, at the end of the sequence, participants showed an adequate understanding of the OH notion. 
They referred to the relationship between human, animal and environmental health, and also provided examples 
of how zoonoses emerge and of the influence of human actions on their emergence. 

Another noteworthy issue is that most students proposed immediate mitigation actions to prevent pandem- 
ics, as explained previously. This points to the need to promote FT as a way to encourage students to move in the 
timespan to propose actions that integrate not only personal experiences, but also information of past episodes; 
this requires prospective thinking (Levrini et al., 2021; Mehren et al., 2018). Anticipating the future is an advantage, 
since it helps to take actions that can influence the evolution of facts (e.g. preventing the emergence and spread 
of zoonosis) and assists in the definition of the most potential favorable scenarios. 

It should be considered that the holistic approach of the learning activities implemented in this study dif- 
fered from the one these students were frequently involved, in which OH was not connected to health topics, and 
this might influence the results obtained. Besides, the selection of participants was done for convenience. Further 
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research developed in other groups and countries will be helpful to enlarge the body of knowledge in this line of 
research. 


Conclusions and Implications 


This research intended to explore how upper secondary school students mobilized ST to tackle COVID-19 
based activities. The interpretative analysis of students’ written productions showed that students articulated sev- 
eral ST aspects, although not consistently. Most students improved their OH notion after completing the activities, 
being able to identify components, relationships, and provide examples of causal relationships. As the results of 
this study suggested, using an OH approach can be a suitable way to engage students in fostering and applying 
ST skills, although further studies are required to provide more evidence to support this. The activity sequence 
could be revised and introduce modelling strategies, as representations facilitate students to address the cognitive 
demand of reasoning with systems. 

Promoting the OH notion and ST-related skills can improve students’ health literacy, enabling them to take 
informed actions. The sequence presented in this work may allow students to better understand health as a global 
issue even when OH is not explicitly mentioned. For instance, it can enable students to learn how environmental 
changes, such as habitat fragmentation due to urbanization or biodiversity loss, increase the frequency of contacts 
between animals and humans, thus increasing the likelihood of transmission of infectious diseases between them. 
This approach would give special emphasis to the importance of taking action to maintain the health of animals, 
humans, and ecosystems. 
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Abstract. One of the skills needed in the 
21st century is critical thinking skills that 
need to be developed for secondary school 
students. Currently, upper-secondary 
students’ critical thinking skills are still 

low. To overcome the problem, website- 
based E-module learning media was 
developed using a STEM approach with 

a Problem-Based Contextual Learning 
model with virtual experiments using PhET. 
The aim of developing this website-based 
E-Module media is to improve students’ 
thinking skills on thermochemistry learning 
materials. The development model is 4-D. 
Data collection in this study was gained 
from three schools representing schools in 
the high, medium, and low categories in 
Surakarta, Indonesia. In this research,216 
students were involved as the subjects at 
the development module implementation 
stage. The results of expert validation of the 
material, language, and media modules 
are suitable for use with a score of 90.03% 
categorized as very good. In the feasibility 
test, the score was 87.98% categorized as 
very good. The score for critical thinking 
skills was 79.18% in the critical category. 
N-Gain score was 0.70 categorized as high 
criteria. The result of the independent 
sample t-test sig 0.023, Critical thinking 
skills were conducted using E-Module. The 
results are higher than conventional media 
and effective in improving students’ critical 
thinking skills. 

Keywords: problem-based contextual 
learning, STEM, e-module, critical thinking 
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Introduction 


Learning is an important foundation in developing individuals’ poten- 
tial and preparing them to face all challenges in the future. The advance 
of technology in this digital era can change the way we learn. Therefore, 
learning approaches need to continue to be adapted to take advantage 
of technology and expand access to global knowledge. So there needs to 
be an appropriate balance between the use of technology and developing 
students’ skills in the 21st century. Learning in the 21st century is focused 
on activities that can train students’ skills by directing a learning process. 
In realizing the learning process for students, teachers must make efforts 
such as providing guidance, encouragement, direction, and good stimuli 
in learning (Mardhiyah et al., 2021). The learning process must be active, 
productive, and meaningful for students, so educators need to be able to 
develop learning approaches. Besides, teachers should emphasize not only 
knowledge and the ability to understand the learning materials but also pay 
attention to affective and psychomotor skills (Sj et al., 2021). In 21st-century 
learning, more emphasis is placed on problem-solving, with critical thinking 
(CT) skills. In 21st-century learning, several skills need to be developed. One of 
the important skills is critical thinking skills (Atabaki et al., 2015; Ennis, 2011; 
Wilson, 2017). Moreover, by having 21st-century competencies, students 
can differentiate between true and false information obtained from the 
environment. This capability is a basic and important competency. CT skills 
are very relevant to the advancement of civilization as they are considered 
the crucial factors foundations of economy-based knowledge so they are not 
only relevant for individuals (Norman et al., 2017). In the world of education, 
CT skills have significantly improved students’ skills and become an ideal way 
to direct and educate children’s and adolescents’ minds (Paul & Elder, 2014) 
and make it easier for students to connect effectively and solve complex 
problems (Basri et al., 2019). Currently, an important goal of the education 
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system in Indonesia is the teaching and application of CT learning (Radulovi’c & Stan’ci’c, 2017; Shamboul, 2022; 
Tiruneh et al., 2017). Teachers, researchers, and policymakers around the world now consider that CT skills need 
to be developed for modern society because they are important and a top priority in self-government societies 
(Ahern et al., 2019; Dumitru et al., 2021). The importance of CT skills is to help students achieve better results in 
solving problems faced in life (Hitchcock, 2021), help students have the ability to adapt to circumstances and to 
be intelligent (Halpern & Diane., 2021), have a good academic achievement (Ren et al., 2020). Apart from that, the 
principles of transformative learning are also in line with 21st-century learning, where students are expected to 
have CT skills and be able to collaborate and solve problems. Accordingly, the role of teachers is very much needed 
in this activity (Rahmawati, 2018). Apart from that, students also need to have CT skills to solve complex problems 
because these skills are needed when entering the world of work (Anderson et al., 2020). Currently, Indonesian 
students have low CT skills. This can be seen from the results of Trends in International Mathematics and Science 
Study in 2018 showing that Indonesian students’ science scores are ranked 45th out of 50 countries (Utomo, 2018). 
Besides, the PISA Assessment for the last 18 years shows that Indonesia is ranked in the bottom 10, far behind many 
other countries since Indonesia's first participation in 2000 (Hewi & Shaleh, 2020). This shows that students’CT skills 
are categorized as very low. In line with research conducted by Irfana et al., (2022) at SMA N 8 Mandau showing 
that students have low critical thinking skills with an average score of 61.2 in chemistry. Apart from that, research 
conducted by Affandy (2019) found that the CT skills of students at SMA Batik 2 Surakarta are still relatively low in 
all indicators. From these problems, efforts are needed to improve students’ CT skills. To improve critical thinking 
skills, an effective learning model where problem-based learning can improve critical thinking skills is needed 
(Park & Choi, 2015). 

To improve students’ thinking skills, as facilitators, teachers must choose an effective learning model because 
an effective learning model can create a complete, interactive, and creative learning atmosphere that can make 
it easier for students to master the learning materials. During the learning process, students can think more criti- 
cally in responding to problems, social skills, and learning outcomes to be optimal. Therefore, the right learning 
process must be used so that the learning process can be successful. To overcome this problem, an appropriate 
learning model is needed. Problem-based learning can improve students’ learning attitudes, critical thinking in 
decision-making, and assessment of subfield problem-solving skills (Park & Choi, 2015; Saepuloh et al., 2021). Apart 
from that, teachers and students can use learning media as their communication tools in the learning process. 
Teachers use the learning media as a technical intermediary and become a method in the learning process so that 
learning activities take place effectively in the classroom (Kuswanto & Radiansah, 2018). The choice of media used 
for learning is adjusted to the situation, conditions, and characteristics of the learning materials (Abidin, 2017). 
Students are considered to still have difficulty understanding chemistry learning materials. This is reinforced by 
around 59.7% stating difficulties in studying chemistry. One material that is considered difficult is thermochemistry. 
Thermochemical learning materials are abstract and difficult to understand, especially those related to concepts 
such as environmental systems and materials (Siagian & Yasthophi, 2021). 

The development of the use of learning media in the field of education is very rapid. One of which is the use 
of learning media modules which have developed into electronic ones called E-Modules. According to Elvarita 
et al., (2020), the e-module developed includes methods, media, and assessments given to students carried out 
periodically and systematically to achieve the competency learning objectives that students must have with their 
respective levels of difficulty. Virtual experimental methods need to be supported by the use of appropriate me- 
dia. Physics and Education Technology (PhET) contains visualizations of real laboratory components that can be 
explored in real-time, concretely, and without risk of injury (Dina, 2020). Another advantage of the module is that 
students can study it individually. To make it easier to access the module, it is displayed on the Google site. The 
advantage of using Google Sites is that it is integrated with other Google applications, such as Google Forms, You- 
Tube, and other Google applications (Nugroho, 2021). The module requires the right learning approach to function 
well. Approaches are the learning methods used by teachers in learning. These learning methods, strategies, and 
models not only help students understand the learning material but also help teachers provide learning services 
in a pleasant atmosphere (Masitah & Sitepu, 2021). According to (Hidayati et al., 2019) STEM increases mastery of 
technology when applied to make learning media (Yusuf & Ahmad, 2020). Learning and approaches using STEM 
can improve students’ CT skills with N-Gain results in the medium category, namely 0.37 (Retnowati, et al., 2020), 
and in Davidi’s (2021) research, the N-Gain results were 0.45. There are three important points in STEM education: 
(1) eliminating barriers between scientific disciplines, (2) being able to integrate between scientific disciplines, and 
(3) placing learning experiences relevant to real-world problems in everyday life. The STEM approach can develop 
science and technology, which are important goals in 21st-century learning (Smith et al., 2022). By learning to solve 


740 


https://doi.org/10.33225/jbse/24.23.739 


Journal of Baltic Science Education, Vol. 23, No. 4, 2024 


ENHANCING GRITICAL THINKING SKILLS THROUGH STEM PROBLEM-BASED CONTEXTUAL ISSN 1648-3898 | /erint, 
LEARNING: AN INTEGRATED E-MODULE EDUCATION WEBSITE WITH VIRTUAL EXPERIMENTS 
eagle ces ISSN 2538-7138 sonine/ 


problems related to complex real-world problems, students have experience outside and inside school. Besides, 
the STEM education approach can provide relevant experiences and stimulate students’ thinking (Murphy et al., 
2019) so that students can build relationships of relevance between STEM disciplines and problems that exist in 
the real world (Smith et al., 2022). Implementing active learning is essential to most integrated STEM approaches 
and has been shown to improve student learning outcomes (Wilson, 2017). 

The application of STEM in Asia not only focuses on expanding STEM-related subjects but also on students’ 
interests in the 21st century including real-world problem-solving, systematic thinking skills, and higher education 
(Lee et al.,2020). In implementing STEM learning, it is necessary to use learning models and learning approaches 
(Suratno et al., 2020). Research conducted by Sara¢ (2018) shows that STEM education is effective in improving 
student learning outcomes, increasing students’ motivation, attitudes, and academic learning achievements, 
and improving problem-solving skills. STEM research is increasingly important and has become an international 
research field so STEM-related research is needed globally (Li et al., 2020; Li, 2019). Contextual learning is the 
way teachers present their everyday life experiences in their classrooms. This aims to help students understand 
the learning material provided by the teacher in the learning process more easily (Aliyyah et al., 2020). The STEM 
approach is in line with PBL which focuses on everyday life problems that are appropriate to real-world contexts 
that build skills, knowledge, resources, and problem-solving practices (Savery, 2019). The four aspects of the STEM 
approach have a harmonious unity with the Problem-Based Learning (PBL) model because they are related to real 
problems so when combined they can improve CT skills (Selisne et al., 2019). This can be integrated because these 
two models and approaches start with problems (Syukri & Ernawati, 2020). In the implementation of PBL learning, 
PBL has several shortcomings including uncertainty about the right learning needs, the depth of knowledge to 
be achieved as well as selecting the right literature, the time required, and choosing the wrong learning needs 
because the teacher does not direct it well (Vanishree & Tegginamani, 2018). Therefore, the presentation of PBL 
learning can be overcome by integrating it with CTL because CTL learning can reduce the weaknesses of PBL, 
namely increasing students’ ability to carry out student activities to carry out observation activities, ask questions, 
and propose hypotheses before carrying out investigative activities. The Contextual Teaching and Learning (CTL) 
approach can encourage students’ critical thinking in which there is a syntax for observing problems, meaning 
identifying them and improving the analysis process of the problems presented but also encouraging their criti- 
cal and high-level thinking (Nawas, 2018; Rahayu, 2017)"type”:"article-journal’”’volume”:"4"},uris”:[“http://www. 
mendeley.com/documents/?uuid=7e3c8297-b84b-40ca-ac6c-cfbd9d355792"}},{"id’:" ITEM-2""itemData”:{“DOI”" 
:”10.1063/1.50160187ISBN”:"97807354160317"ISSN”:"15517616" abstract”:"Students need to be equipped with 
the 21st century skills/capabilities to ensure their competitiveness in the knowledge era. So, it is imperative that 
education at school should be changed in order to fulfill the need. However, there is not any specified approach on 
how to educate young students for the 21st century capabilities. Regardless the impediment for ts exist, we need 
to construct an innovative instruction that can develop the students’ 21st century skills by incorporating the skills 
needed, based on contemporary theory of learning, necessary context of learning and appropriate assessment in 
a chemistry subject matter. This paper discuss the feasible skills to be promoted through chemistry course. Those 
skills/capabilities are scientific literacy, higher order thinking, communicationand collaboration and curiosity. The 
promoted are called the 21st century scientific literacy skills in which it emphasis on scientific literacy and embed- 
ded the other 21st century skills into the innovative chemistry instruction. The elements involve in the instruction 
such as inquiry and constructivist approach, nature of science, contemporary/socioscientific issues, critical think- 
ing (higher order thinking. The CTL strategy is very important because CTL emphasizes critical thinking and can 
recognize teaching and learning needs. The CTL approach can motivate students to learn from each other and use 
an evaluation process (Hakim et al., 2023). 

The integration of PBL and CTL is carried out by combining the syntax between Problem-Based Contextual 
Learning to be: 1) Problem-oriented; 2) Organizing students; 3) Making observations; 4) Proposing a hypothesis; 5) 
Carrying out investigations; 6) Development and presentation of results; 7) Analysis and evaluation. The integra- 
tion between PBL and CTL (Problem-Based Contextual Learning) can improve students’ critical thinking abilities 
(Husna et al., 2019). STEM-integrated education allows teachers to convey science, technology, engineering, and 
mathematics content and relate it to real situations (Sumarni & Kadarwati, 2020). A learning method known as the 
STEM approach blends science, technology, engineering, and math (Tasdemir, 2022) which can improve students’ 
CT and problem-solving skills (Dogan & Kahraman, 2021) (Dogan & Kahraman, 2021) so that this approach can 
improve CT skills by integrating the PBL learning model and integrating CTL with STEM. 

In this research, an E-module is developed which aims to overcome students’ low CT skills. The developed 
E-Module can be accessed by students using a smartphone or laptop allowing them to learn independently and 
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flexibly in terms of time and place. The developed E-Module contains several elements that are arranged based 
on the STEM-Problem-based contextual learning model and approach. In the arrangement, there are animations, 
videos, images, and interactive features such as virtual laboratories using PhET. Apart from that, the developed 
E-module not only has interactive content but also has problem-based challenges and exercises, formative evalua- 
tion, independent exploration, discussion, and collaboration, as well as assessment and reflection that can improve 
students’CT skills. Elements in the E-module that are very influential in increasing students’CT are presenting ques- 
tions to solve problems. In the Problem-based contextual learning syntax, students are asked to solve problems 
related to their day-to-day life problems (Sugianto et al., 2013). The developed E-Module also provides students 
with the opportunity to hold discussions and collaborate with groups to solve problems using students’ worksheets 
(LKPD) and carry out integrated formative assessments during the learning process (Aulia & Hardeli, 2022). The 
interactive content in the E-Module includes animations, videos, and simulations which are displayed attractively 
and can improve students’ CT skills in using the E-Module (Kurniawan et al., 2018). 

The results of field analysis during learning conducted on 144 students showed that 59.7% of students stated 
that chemistry was a difficult subject. Initial analysis of students also showed that 60% thought chemistry was ab- 
stract and, therefore, it is difficult to understand, 13% of students thought the teacher's learning material delivery 
was less interesting, and 10% of students said the learning media was lacking. The students’ chemistry scores were 
43% (fair), 30% (poor), 20% (good), and 2% (very good). Problems in learning chemistry experienced by students 
include abstract chemical material, difficulty in understanding formulas, uninteresting teachers’ teaching strategies, 
abstract and complex learning materials, lack of learning media, and student visibility problems. 

Based on the problems presented above, the researchers conducted research regarding the development 
of E-modules using the Google site. The E-Module was developed using a STEM approach with a PBL model inte- 
grated with Contextual Teaching Learning (Problem-Based Contextual Learning). The media developed is equipped 
with virtual experiments supported by the PHeT Colorado application included in Google Sites. This module was 
developed so that upper secondary students’ CT skills in thermochemical materials increase. 

The research questions were formulated as follows: 

1. How to create a chemistry learning e-module based on STEM Problem-Based Contextual Learning 
supported by Google Sites and PhET Colorado applications? 

2. What is the feasibility of a chemistry learning e-module based on STEM Problem-Based Contextual 
Learning to enhance students’ capacity for critical thinking in chemistry learning material? 

3. How is the effectiveness of STEM-Problem Based Contextual Learning based chemistry learning in 
improving students’ critical thinking skills in thermochemical learning material? 


Research Methodology 
General Background 


This research was developed using the research and development (R&D) method to create a product that was 
tested for efficacy after it was developed. The outcome of this research was website-based learning materials using 
a STEM-Problem-Based Contextual Learning methodology, which aims to incorporate STEM into the classroom and 
address students’ deficiencies in critical thinking. The product development of this study makes use of Thiagarajan 
et al., (1974) 4D research model. The design of the web-based STEM E-Module platform is presented in Figure 1. 

The first stage was to conduct a survey of 144 students using a questionnaire. Apart from that, a survey was 
also conducted on teachers using a questionnaire. The purpose of conducting a preliminary analysis is to find out 
the learning process that has been carried out by the teacher, to know the problems experienced, the desired 
module content requirements, the interest of teachers and students in STEM learning, and the availability of school 
facilities and infrastructure to implement electronic modules including Front-end Analysis, task analysis, learner 
analysis, concept analysis, and determining instructional goals. 

The second stage is the process of designing website-based learning media. Currently, creating learning 
materials in the form of e-modules is the goal. The selection of e-module development is in accordance with the 
students’ learning objectives, material, characteristics, and needs. Activities carried out included initial product 
design, planning learning objectives, compiling instruments (module validation, critical thinking, and learning 
implementation), and the stages of criterion-referenced test construction, media selection, format selection, and 
initial design. 
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In the third stage, namely the development stage, validation was carried out by material experts, media ex- 
perts, language experts, and practitioners (teachers). After all the media were valid, trials were carried out in small 
groups and large groups on all aspects developed including media aspects, linguistic aspects, material aspects, 
learning aspects and media use aspects. 

The final aspect was to determine the effectiveness of the Web-based E-Module platform with the Problem- 
Based Contextual Learning approach. At this stage, an integrated comparison of experimental class and control 
class data was carried out. Pretest data and initial observations of critical thinking obtained in the design step 
were taken. The learning time for thermochemical material is three meetings. During the learning process, obser- 
vations were made using critical thinking and the implementation of the learning by the observer. After learning 
was completed, a posttest was carried out. After that, an N-Gain analysis was carried out to see the increase in the 
results of initial and final observations of critical thinking to find out how effective the module was in enhancing 
students’ capacity for critical thinking. In this research, the researchers reported research up to the media develop- 
ment stage with improvements made at each research stage. 


Figure 1 
Research Procedure 


Problem Development goal 
Analysis Module 


Define 


Initial 
design of 
module 


Module 
Formatting 


Module 


Development Validation 


Development 


Large group Draft Revisi 
trial Product 2 en 


Dissemination 


E-Modul STEM 


bats Thermodynamics 


Participants 


The research was conducted on class 11 thermochemical learning material at three schools in Solo, Central 
Java. The entire process was carried out during the 2023-2024 academic year. The web-based STEM-Problem Based 
Contextual Learning E-Module learning media development stage involved 30 students and the wide-scale test- 
ing required 108 students. The development was followed by a focus group discussion consisting of four lecturers 
and two teachers who are experts in the fields of media, language, content, and STEM. Assessment practitioners 
of 7 experimental class chemistry teachers were also used at the development stage of the STEM-Problem Based 
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Contextual Learning E-Module learning media. The final stage involved 216 students who were divided into ex- 
perimental classes and control classes. All students are first-semester 1 1th-grade students from 3 state schools in 
Surakarta, Central Java, Indonesia. Sample characteristics are presented in Table 1 


Table 1 
Characteristics and Distribution of the Sample 


Variable Control Class Experimental Class 
Range Age 16-17 16-17 
Female 68 81 
Gender 
Male 27 40 
Total = 216 108 108 
Instrument 


The development of the web-based STEM - Problem-Based Contextual Learning Chemistry E-module was 
designed by conducting a preliminary survey on 144 students and 7 chemistry teachers. The survey with a ques- 
tionnaire consisted of 6 aspects,16 indicators for students, 10 aspects for teachers, and 32 indicators regarding 
chemistry learning carried out by teachers. 


Table 2 
Research Instruments 
Stage Instrument Subject Time 
Needs Analysis Questionnaire Student and Teachers Before product development 
Module Validation Questionnaire Experts, education practitioners, Before product try-out 
and peers 
Module Readability Questionnaire Students After the learning process 
Level 
Critical Thinking skills Critical thinking questions on thermo- Students Before and after the learning 
chemistry learning materials process 


There are ways students learn chemistry, student assessments of chemistry teaching-learning activities (PBM), 
student assessments of the materials/textbooks used by teachers, student assessments of LKPD, students’ readiness 
to accept research products, teacher STEM questionnaire attitudes towards STEM education, STEM knowledge, 
the utilization of technology and science, engineering and science, science-mathematics, science-technology- 
engineering, and technology and science mathematics applications (Budiyono et al., 2020; Suratno et al., 2020). The 
module validation instrument used referred to Prastowo (2013) including media validation consisting of content 
quality aspects with 13 indicators and instructional quality with 8 indicators, material validation consisting of visual 
communication aspects with 5 indicators, and technical aspects with 4 indicators as well as language validation 
consisting of communicative aspects consisting of 2 items, and aspects of term accuracy consisting of 2 indicators. 
Then, in the application aspect of learning media, CT skills activities were carried out with five dimensions measured, 
including interpretation, analysis, evaluation, inference, and explanation. Observation sheet on implementation of 
STEM-Problem contextual learning with six learning steps consisting of problem orientation, organizing students, 
observation, hypothesis, investigation, and presentation of results and conclusions. The instruments used in data 
collection were validated by 6 teachers. 


Integration E-Modul STEM -Problem-Based Contextual Learning based website 


The website-based STEM-Problem Based Contextual Learning E-Module was applied to students in class XI 
in their first semester in the 2023-2024 academic year on thermochemistry learning materials. Thermochemical 
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learning material taught included chemical reactions and enthalpy changes, types of enthalpy changes, calorim- 
etry, and Hess's law. Learning activities included projects that students must work on. The time allocated for topics 
in the control and experimental classes can be seen in Table 2. Each learning activity was carried out over 1 week 
with an accumulation of 5 hours of lessons (1x45 minutes). Thermochemistry learning using the website-based 
STEM-Problem Based Contextual Learning E-Module was only taught in experimental classes. There was a discus- 
sion about how to integrate aspects of science, technology, engineering, and mathematics in chemistry learning. In 
presenting learning outcomes, students can present them in several forms, using PPTs, e-posters, and infographics). 


Table 3 
Division of Learning Time Between Experimental Classes (EC) and Control Classes (CC) 


Topic 1 (Thermochemical reactions and Topic 2 (Types of Topic 3 Topic 4 
pas enthalpy changes) (enthalpy changes) (Calorimetry) (Hess’s Law) 
Week 1 Week 2 Week 3 Week 4 


EC: Experimental Class; CC: Control Class 


Data Collection and Analysis 
Requirements analysis data (Module Characteristics) 


This stage was carried out to find out whether the module meets the characteristics that must be fulfilled as a 
learning medium. Descriptive analysis was carried out on the data resulting from the needs analysis. The E-module 
was designed according to development in accordance with the initial analysis, namely by developing a STEM- 
Problem--based Contextual Learning E-module, which was integrated with website-based Contextual Teaching 
Learning with Google Site and equipped with simulations using PhET. 


E-Module development data (Module Feasibility) 


The feasibility of the developed module was seen from the assessment carried out by expert validation, 
including language, media, and materials experts, in addition to the evaluation of chemistry instructors. Student 
assessments were carried out during small-scale trials and large-scale trials in terms of language, media, materials, 
usability, and learning. After the data was obtained, they were processed into scores which were categorized into five 
criteria according to Table 2. A module is said to be feasible if it is at least in the ‘sufficient’ category (Azwar, 2007). 


Table 4 
Categories of Module Feasibility Assessment Validation 


Interval score Criteria 
Mi+ 1,5 Sbi< X Very Good 
Mi+ 0,5 Sbi< X < Mi+ 1,5 Sbi Good 
Mi - 0,5 Sbi < X < Mi+ 0,5 Sbi Medium 
Mi - 1,5 Sbi< X < Mi - 0,5 Sbi Low 
X < Mi-— 1,5 Sbi Very Low 
Information : 
Xx = Respondent’s score 
Mi = Ideal Mean 
Sbi =TIdeal standart deviation 
Mi = 4 (ideal maximum score + ideal minimum score) 
Sbi = J6 )(ideal maximum score - ideal minimum score) 
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Feasibility was also obtained from observing the implementation of learning when the module was tested. 
The observation results were converted into percentages and then grouped into four categories of learning flow 
implementation according to Yamsari (2010) in Table 3 

Score (Percent)= (Total Score)/(Maximum Score) x 100 % 


Table 5 
Learning Flow Implementation Criteria 


No Total Score Percentage Implementation Criteria 
1 86 - 100 Very well executed 
2 71-85 Done 
3 51-70 Less implemented 
4 Less than 50 Not implemented 


Field try-out data (Module Effectiveness) 


Next, to find out whether the module created was successful in enhancing CT skills, posttests were utilized 
to evaluate the improvement of critical thinking abilities. The results of the post-test work analysis were analyzed 
using Normalized Gain (N-Gain). According to Hake (1999), learning is categorized into the effective category if 
the N-gain obtained meets the criteria: 


Table 6 
Category N-Gain Value 
No Interval Score Criteria 
1 <07 High 
2 0.7>x>0.3 Medium 
3 <0.3 Low 


E-Module operational field test analysis 


The data analyzed in the field test included critical thinking ability analysis data. The aim was to see the sig- 
nificance of the influence of learning using the website-based STEM Problem Based Contextual Learning E-Module, 
which is equipped with virtual simulations/experiments on thermochemical material. Paired sample t-test is the test 
that is employed if the data is normal and homogeneous. The effectiveness of the data was tested by calculating 
the increase in learning outcomes tested using HOTS questions with development based on measures of critical 
thinking abilities using the normalized gain technique. 

The hypothesis of this research is: 

H, = Critical thinking skills of students who use E-Module media are the same as those classes that use 
conventional media 
H, = The critical thinking skills of students who use E-Modul media are higher than students who use 
conventional media 
In research, Ho is accepted if it has a significance value below .05 (p < .05). 


Research Results 
Define Stage 
At the definition stage, data was obtained from a survey conducted on 144 students using a questionnaire. 


Apart from that, a survey was also carried out on 8 teachers using a questionnaire regarding learning carried out 
at school in Table 5. Data obtained in Table 6 were obtained. 
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Table 7 
The Respondents of Needs Analysis 


Gender 
Number of Respondent — Tl Age 
Male Female 
144 58 86 16-17 
Table 8 
Result of Student Questionnaire Needs Analysis 
No Statement Percentage (%) 
1. Chemistry is a difficult subject 60 
2. Chemistry learning is abstract, so it is difficult to understand 60 
3. Teachers do not use student worksheets for chemistry learning 58 
4. Teachers do not use virtual laboratories/do online simulation practices in chemistry learning 60 
5. Teachers do not use E-Module in chemistry learning 42 
6. Using visualization media will increase your understanding of chemical materials 82 


Based on the results of the initial analysis of students, the results were obtained on what models or learning 
strategies teachers usually used in chemistry subjects, depending on the findings of surveys, namely lectures, 
questions and answers, and discussions. According to students, an interesting strategy is based on the results of 
analysis using discussions, questions and answers, and games to increase interest in learning, direct explanations, 
and scaffolding by the teacher. Ownership of books/teaching materials depending on the findings of surveys is 
not required to have a book handle. From the questionnaire, 27% of students thought their textbooks were less 
interesting, and 1% of students thought their textbooks were less interesting. Regarding the ownership of student 
workbooks (LKS), 58% of students did not have LKS. Based on this analysis, it implied that students required easy 
and interesting learning media in learning chemistry. Visualization media was needed because students considered 
chemistry to be an abstract lesson, so they needed to use visualization simulations in chemistry learning. Learning 
using the lecture method made students less interested, and their abilities were not maximally honed. This can be 
seen from the grades obtained by students in learning chemistry. So, it is very important to develop E-Modules 
based on Problem-Based Contextual learning with a STEM approach to improve 21st-century skills, one of which 
is critical thinking. 


Front-end Analysis 


Based on the initial needs analysis carried out on 144 students, it can be concluded that students needed 
easy and interesting learning media in learning chemistry. In school learning, no adequate teaching materials are 
available for the learning process, and in practice, the media used by teachers in learning is still conventional. The 
media could not accommodate all students’ learning styles. Based on the initial needs analysis, it can be concluded 
that students needed easy and interesting learning media in learning chemistry. Media visualization was needed 
because students considered chemistry to be an abstract lesson, so they needed to use visualization simulations 
in chemistry learning. Learning using the lecture method made students less interested, and their abilities were 
less honed. This can be seen from the grades obtained by students in learning chemistry. The ability that students 
must have in the 21st century is the ability to think critically. Where these abilities can be improved by developing 
learning media with the Problem-Based Contextual Learning model with a STEM approach which can improve 
students’ CT skills in learning chemistry. 


Learner Analysis 


At this stage, student analysis was carried out through a student needs questionnaire. Students were consid- 
ered to still have difficulty understanding chemistry material. This is supported by the analysis’s findings of student 
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needs revealing that around 59.7% expressed difficulties in learning chemistry. One material that was considered 
difficult was thermochemistry. This is in line with research conducted by Toshtemirovna (2023) stating that some 
students had significant difficulties in mastering chemistry. This can be overcome with technology so that it can 
increase the strength of chemical knowledge. 


Task analysis 


Currently, a review of the learning objectives and minimum criteria of mastery learning was carried out. The 
review was then translated into several indicators that students had to achieve. The results of the initial analysis 
showed that the learning objectives of students were still low, as evidenced by the grades obtained by students, 
which were still below the criteria. This was consistent with studies carried out by (Rahmi & Azra, 2023)83.69% of 
SMAN 13 Padang class XI MIPA students still need to achieve the minimum completeness criteria set. It indicates 
that students have learning difficulties. Learning difficulties are failures to attain learning goals characterized by low 
learning outcomes. This study aims to determine the percentage of students who experience learning difficulties 
for each indicator on thermochemistry material and determine the factors that cause learning difficulties experi- 
enced by students in terms of learning methods. This research is a type of descriptive research. The sample for this 
research was students in class XI MIPA 6 at SMAN 13 Padang in the 2022/2023 academic year, which consisted of 34 
people. The research instrument was a diagnostic test with four-tier multiple-choice questionnaires and interviews. 
The data analysis used is descriptive, namely analyzing and providing an understanding of the data in the form of 
numbers so that an overview can be given in an orderly, concise, and transparent manner. The results of this study 
stated that students of SMAN 13 Padang had difficulty learning thermochemistry material in the high category. The 
highest difficulty level is found in the sixth indicator (determining the AH value of the reaction through a calorim- 
eter experiment which was conducted at 2 state schools in Padang and found that chemistry learning outcomes 
of 83.69% were still below the minimum criteria of mastery learning for thermochemical learning material. Apart 
from that, according to studies carried out by Zakiyah et al., (2018)the influence of student’s\nunderstanding of 
stoichiometry on their learning outcomes of thermochemistry, the effects of \nremidial teaching using problem 
solving method in stoichiometry and in thermochemistry on \nthe students understanding of thermochemistry, 
and the effectiveness of both remidial \nmethods on student's learning outcomes of thermochemistry. This re- 
search used descriptive \nand quasi experimental designs. The subjects were students of year XI of SMA Negeri 2 
\nPamekasan of the academic year 2014/2015. Data collecting procedure was done by a test \nmethod and tests 
of hypothesis were done by t-test procedures (a of 0.05 and Sirhan (2007) chemistry was a difficult material, one 
of which was thermochemical learning material because in thermochemical material there were abstract concepts 
and required mastery of mathematics and a strong memory. 


Concept Analysis 
After carrying out the task analysis, a concept analysis was conducted. At this stage, a concept analysis, which 


was used to determine the learning concepts that students need to master regarding thermochemistry learning, 
was Carried out. 
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Figure 2 
Arrangement of E-modules Developed 


E-Module media based on the Google 
site for Thermochemistry material 
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Home page Introduction Learning Reflection Evaluation 
Activities 
| 
| 
Contains Contains Contains problem- Contains Berisi soal evaluasi 
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about the E- identity, concept activities, organizing learning evaluation 
Module being map, CP, TP, students, observation, questions 
developed KKTP, material, hypothesis, 
keywords, investigation, 
syntax and presentation of results 
instructions for and conclusions 
use 


Specifying Instructional Objectives 


At this stage, the learning competency objectives that students must achieve were determined. This research 
and development aimed to encourage, help, and facilitate students to comprehend thermochemical material 
correctly. 


Table 9 
Observation Results of Students’ Initial Critical Thinking Skills 


Criteria of Critical Thinking Skills 


No Representative Schools Twa Total Student 
Very Low Low Medium High Very High 
1 High Category 20 39 6 7 0 72 
2 Medium category 34 20 12 6 0 72 
3 Low Category 33 22 13 4 0 72 
Table 10 


Student Pretest Results 


Pretest Results Criteria 


No Representative School = — ——_-_—2-----YNA Total Student 
Very Low Low Medium High Very High 
1 High Category 16 34 17 5 0 72 
2 Medium category 14 27 23 8 0 72 
3 Low Category 21 34 14 3 0 72 


The results of observations of students’ important initial thinking abilities can be seen in Table 7. It is evident 
that the typical students from the three schools were in the uncritical category. Based on the pretest results in the 
three schools, it can be seen in Table 8 that the average of students in the High Category, Medium Category, and 
Low Category in Surakarta is not critical. This is consistent with studies implemented by Khoirunnisa & Sabekti 
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(2020), which stated that the critical thinking skills of students in the city of Tanjungpinang on chemical material, 
namely chemical bond material, were very low. Apart from that, research conducted by Fernanda et al., (2019) on 
buffer solution material showed that students still experienced difficulty in making observations, making deci- 
sions, making inductions, identifying assumptions, and experiencing difficulty in finding credible learning sources, 
observing, and making decisions. 


Design Stage 
Constructing Criterion-Referenced Test 


Designing instruments used for research was conducted in this stage. The E-Module product assessment in- 
struments were in the form of student needs analysis questionnaires. Analysis was needed for teachers regarding 
STEM, pre-test and post-test critical thinking questions, module validation instruments, critical thinking skills activity 
instruments, and learning implementation observation instruments regarding Problem-Based Contextual Learning. 


Media Selection 


The advantage of the module was that students were able to study it individually. To make it easier to access 
the module, it was displayed on the Google site. The advantage of using Google Sites is that it is integrated with 
other Google applications, Google Forms, YouTube, and other Google applications (Nugroho, 2021). 


Format Selection 


The appearance and content design in developing E-Module learning media included several home menus, 
introductions, learning activities, reflection, and evaluation. Each menu in this application had its own function. 
More detailed contents of this research and development product can be seen in the following chart in Figure 2. 


Initial Design 


The design stage was the stage of designing and preparing the E-Module media that was created. The media 
menu page contained 5 menus consisting of the home menu, introduction, learning activities, reflection, and 
evaluation. The activity menu contained 4 main topics related to Thermochemistry and enthalpy changes, Types 
of enthalpy changes, Calorimetry, and Hess's Law. 


Table 11 
Electronic Module Design 


No Module Section Explanation of Module Section 
This section contains the title, UNS logo, module target subject, module compiler, and module compiler 
1 Home page 
agency. 
2 Material Identity This section contains subjects, class/semester, main material, sub-main material, and time allocation. 
This section contains an illustration of a chart that connects one concept to another concept in thermo- 
3 Concept maps : 2 
chemical material. 
. Learning outcomes (CP) are competencies that students must be able to achieve in the learning 
4 Learning Outcomes 
process. 
: er Learning objectives are students’ achievements in the competency aspect which includes knowledge, 
5 Learning objectives ; ‘ : : : sae 
attitudes, and skills that students can acquire during learning activities. 
KKTP is a series of criteria or indicators that show the level of students’ ability to achieve a certain 
6 KKTP : ; a 
competency in the learning objectives that have been set. 
7 Material This section contains scientific material regarding thermochemistry. 
Keywords are words or phrases that stand out in the title, header, subject, abstract, or text, as well as 
8 Keywords 


content notes. 
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No Module Section Explanation of Module Section 


9 Leaming Syntax This section contains an explanation of the learning flow according to the problem-based contextual 
learning syntax, which contains STEM components. 

10 Probient Onentation Section cararuoatela aca pet a the logistics required (convey problem orientation), 

11 Part of Organizing Students Motivating students 

12 Otnaon sein Tues sap ne os rie sin each go 

13 Hypothesis Section This section guides students in creating hypotheses 

14 Investigation Section This section guides students in creating hypotheses 

15 Results Presentation Section This section guides students in collecting data or information, compiling data, and analyzing them. 

16 Conclusion section This section invites students to make presentations 

17 Reflection Together with the students, teachers draw conclusions and reflections. 

18 Evaluation This section contains 2 types of HOTS for each activity 

19 Assessment Guidelines This section contains HOTS questions 

20 Glossary This section contains scoring guidelines for learning activities 

Figure 3 


Main Menu Display E-Modul 
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Development Stage 


The module's first step in the development process was to validate the product created with language experts, 
media experts, and learning material experts. This research used two media experts consisting of two lecturers, 
material experts consisting of two lecturers, and language experts consisting of two experts. Practitioner research 
was conducted by seven teachers representing state and private schools. 
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Expert's Validation 


Table 12 
Result of Expert Validation 


No Expert’s Indicators Score (%) Criteria Feasibility 


Suitability of content, accuracy of content, complete- 


1 Material ness of content, systematic presentation, student 89.28 Very Good Worth to use 
motivation 

9 Media Neatness, accuracy, readability, navigation, and 87.50 Very Good Worhio use 
convenience 

3 Language Communicative language and accuracy of sentence 93.33 Very Good Waniiouce 


construction 


Table 10 results indicated that the E-module’s validation by media, language, and content specialists revealed 
the material produced was good in terms of content and based on the problem-based contextual learning model, it 
was necessary to add additional learning objectives and learning outcomes. Media experts stated that the product 
was developed well, but a sub-menu on the homepage regarding learning outcomes and learning elements needed 
to be created. The words in the subtitles needed to be simpler, and the arrangement of images and videos in the 
activities needed to be rearranged. Linguists needed to make improvements regarding the writing of instructions 
for learning syntax so that it became more communicative for users. The module can be used in learning after 
improvements have been made. This is in line with research by Yulianto et al., (2009) which stated that for the 
learning media to be chosen appropriately, it is important to be mindful of objectivity, effectiveness, and efficiency 
in the use of learning media, where the objectives were formulated in the teaching materials to be delivered and 
make it simpler for learners. The input provided by experts was revised, and the product was then used for try-out. 


Table 13 
Feasibility of Chemistry E-Module 


No Aspects of Research Results Average (%) Category Feasibility 
1 Teacher’s assessment 95.38 Very good Worth to use 
2 Small group try-out 84.91 Very good Worth to use 
3 Large group try-out 83.66 Very good Worth to use 


Based on Table 13, it can be seen that all aspects of the development of the STEM-Problem Based Contextual 
Learning E-module developed by researchers on thermochemical material had met the product development 
criteria. 
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Table 14 
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The Result of Module Development 


No Module Section 


Module Section Screnshoot and STEM Components on the Module 


1 Problem Science 
Orientation At the problem orientation stage, students can observe the phenomena presented by viewing the videos presented, the 
images provided, and reading the descriptions provided on the E-module page. Students use their critical thinking skills 
to process analysis results. Interpretation of existing phenomena to ensure students have an idea of what students must 
prepare to achieve learning goals. 
Orientasi Masalah 
SATE KERE KULINER KHAS SOLO 
Sate Kere 
2 Organizing 
Students (-) 


Mengorganisasikan Peserta 
Didik 


rta didik, diharapkan peserta didik memehami tujuan pembelajaran yang akan dilakukan dalam kegiatan 1, 
ngan lancar: 


jutny ah pem 


a Jompok. Bu Nanda akan membagi kalian dalam 6 kelompok secara homogen. Silahkan kalian 
n daftar kelompak yang ada 


kelompok 1 


Kelompok 2 


3 Observation 
(Science) 


Science 

Students observe the phenomena presented. The observation process is carried out based on problem discourse or after 
students see the display in the form of provided pictures, articles, or videos. Based on these observations, students are 
expected to be able to identify existing problems and formulate problems (ask questions) related to observations of the 
learning material carried out. 


Observasi 


Selanjutnya kita masuk ditahap Ob: 
ada disekitar kalian yang berhubun; 
kalian melakukan observasi pada goo 


berapa fenomena dan masalah yang 
lis pertanyaan yang muncul ketika 


Reaksi Pembakaran 


J burn 


mia 
adalah reaksi pernbakaran. Misainya saat kita membeli 


4 Hypothesis 
(Science) 


Science 
It is expected that students can analyze and estimate answers or temporary solutions to the problem formulations they 
have made and connect them with previous knowledge through observations. 


nyelesaian yang bersifat 
yang telah dilakukannya dan 
silahkan disubmit pada google 


a” 


a. Tumbuhan dapat hidup dengan melakukan fotosintesis. Dalam melakukan fotosintesis, tumbuhan membutuhkan energi. Energi diserap 
dari cahaya matahari oleh zat hijau daun (disebut klorofil) yang digunakan membentuk glukosa. Glukosa yang dihasilkan diubah menjadi 
pati (amilum) kemudian disimpan dalam berbagai sel seperti biji, akar, umbi, dan juga buah sebagai cadangan energi. Termasuk jenis 
reaksi apakah itu? Bagaimana menuliskan reaksi fotosintesis? 

b. Salah satu contoh reaksi kimia adalah reaksi pembakaran. Misalnya saat kita membell sate bakar, penjual akan membakar sate 

oO menggunakan arang. Jika kita berada di dekatnya, kita akan merasa panas. Termasuk jenis reaksi apakah itu? Bagaimana menuliskan 

reaksi pembakaran arang (dengan asumsi arang hanya berisi karbon)? 
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5 Investigation Science 

(Engineering, Students can collect information by doing practicums, studying literature, or opening websites on the internet. It is 
Science, anticipated that students are ultimately able to find answers to the problems they have formulated and able to determine 
Mathematics) whether or not the hypothesis formulated is true. 

Students conduct experiments using procedures prepared using the PhET Colorado application 

Manipulation 

Students conduct analysis, design experiments, and develop procedures for conducting experiments. 

Mathematics 


Students carry out experimental analysis on thermochemical material which aims to obtain relationships between vari- 
ables. The abilities needed are mathematical skills and critical thinking 


Penyelidikan 


silhkan menegtjakan LKPD yang ada, dan silahkan lakukan simulasi mengenai energi dengan menggunakan PhET 


& Energy Symbols @ 
= © Link Heaters & 
a 


6 Presentation of Science 
results (Science) Students present the results of analytical experiments on LKPD which are carried out using language and constructing 


Penyajian hasil 


sudah kallan kerjakan didepan kelas 


rm yang disediakan 


itt © Wiis 


good sentences 


7 Conclusion (-) 


REFLEKS!| PEMBELAJARAN 


8 Evaluation Science 
(Science, Students can analyze the problems/questions presented and provide appropriate solutions or answers according to the 
Mathematics) understanding they gain during the learning process. 
Mathematics 
Students can solve the questions given by carrying out analysis coherently and correctly. 


Evaluasi 


Silahkan kerjakan soal pada form dibawah Ini 
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Disseminate Stage 
a) Final criteria of critical thinking skills 


Table 15. 
Final Criteria for Critical Thinking Skill Level 


Critical Thinking Criteria 


No ws os fF FFT Tmo _ Total = Mean (%) Criteria 
ene? Very Critical Critical Medium Low Very Low 
1 High Category 38 30 4 0 0 72 79,73 Critical 
2 Medium category 23 37 12 0 0 72 77,56 Critical 
3 Low Category 42 28 2 0 0 72 80,19 Critical 
b) Descriptive Statistical of Pre-Test and Post-Test Critical Thinking Skill 
Table 16 


Descriptive Statistical of Pre-Test and Post-Test for Critical Thinking Skills 


R tati Pre-test Post-test 
No choi ‘ i we Group NG ain 
ene? N — Range X SD Range X SD 
; High Category Experiment 36 64 44.556 14.619 22 85.222 5.329 na 
Control 36 58 46.416 15.329 24 76.805 5.721 
Experiment 36 73 49.861 17.053 26 81.861 6.512 
2 Medium category 0.69 
Control 36 58 54.889 15.555 32 75.500 6.954 
Experiment 36 60 46.138 13.983 32 75.333 6.811 
3 Low Category 0.67 
Control 36 60 45.694 12.363 36 72.555 10.728 


Table 16 shows a summary of the results of the descriptive statistical analysis which reveals that the post- 
test average of the CT skills of the students in the experimental class was greater than the control class's average. 
The standard deviation of CT skills in learning using the STEM Problem Based Contextual Learning E-Module was 
smaller than the standard deviation of the control class. Based on the findings of the research, implementing the 
E-Module STEM-Problem-based contextual learning media could enhance students’ CT skills using an average of 
79.16% in the three school categories from low, medium, and high categories in the critical category. Apart from 
that, it was also supported by the highest average N-Gain results in high-category schools, meaning that CT skills 
using E-Module media were higher when compared to students who made use of traditional media. The N-Gain 
score for students’ CT skills was 0.69 based on the calculated data, including at a high level (Hake, 1999). This was 
consistent with a study by (Lestari et al., 2018), indicating that the application of E-Modules could provide meaning- 
ful learning experiences for students, so using the STEM approach and problem-based contextual learning could 
improve students’ CT skills. The STEM approach can be applied to classroom learning in various forms of activities, 
including as models, teaching materials, or Student Worksheets (LKPD) (Cruz et al., 2021; English, 2017; Lestari 
et al., 2018). The PBL learning model can increase student activity during learning and improve students’ critical 
thinking skills (Yulia & Salirawati, 2023) 
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Figure 4 


Pretest-Posttest N-Gain Value Criteria 


SS 70 
@ 60 
o . 
Bo 50 BN-Gain 
= 40 
g 30 
= ped 
© 2 N-Gain 
10 
ie} 


High Category Mediumcategory Low Category 
Representative Schools 


The results of the pretest and post-test research were used to determine CT skills. It is evident that students 
in the experimental class had better CT skills than those in the control group. These results showed that integrat- 
ing the E-Module based on the STEM-Problem Based Contextual Learning website can be achieved to improve 
students’ CT skills in chemistry learning. 


Table 17 
Results of Critical Thinking Skills Analysis 


No Test Test Type Significance Decision Conclusion 
N-Gain 
‘ ; ; Exp = .265 Ho ae 
1 Normality Shapiro-Wilk N-Gain Control= accepted Normally Distributed Data 
103 
Ho The variance of the chemistry pretest and posttest 


2 Homogeneity Levene’s Test 135 N-Gain data for students in the experimental class 
accepted 
and control class was homogeneous 


Difevents ower These results state that the critical thinking skills of 
Experimental Independent Ho 


N-Gain and Control Sample T-Test 023 was rejected students who use E-Module media are higher than 
students who use conventional media. 


N-Gain results 


Tests for normality, homogeneity, and paired sample t-tests were conducted in this research by using the SPSS 
22 software. In the normality test, because the sample consisted of less than 50 participants, the Shapiro-Wilk test 
was used. In normality testing, the significance value (p) obtained for the experimental N-Gain was .265, and for the 
control N-Gain, the significance value (p) was obtained at .103 with a significance value greater than .05, indicating 
that the students’ CT skills data were assumed to be normal. After that, homogeneity was tested using Levene’s 
test. The result was a significance value of .135 (p>.05), indicating homogenous data. An independent sample t-test 
was carried out to look for differences in students’ CT skills before and after the treatment administration using the 
E-Module problem-based contextual learning. The result was a significance of .023 (p < .05), demonstrating that 
there was an obvious difference in the students’ CT skills. Enhancing students’ CT skills with the media developed 
using the STEM-Problem-based contextual learning approach using learning syntax started from problem orienta- 
tion to a natural phenomenon that was commonly experienced in everyday life. Next, organizing students to form 
groups according to their respective learning styles. Then students made observations by making observations, 
discussing problem identification, and formulating problems from the problems presented so that students can 
develop CT skills by identifying problems. The next step was carrying out the hypothesis stage by finding possible 
alternative answers. Next, an investigation was carried out to find a solution to the problem. At the investigation 
stage, students could complete the project given on the LKPD used to sharpen students’ thinking at the design 
stage. In the final stage, students presented their results and concluded. This was in line with research by Budiyono 
et al., (2020) who conducted the same research with the STEM E-Module to provide an increase in CT skills. In the 
research, learning activity was carried out in the classroom by providing e-modules for discussion. In the experi- 
mental stage, it was carried out virtually using the virtual laboratory application simulation. 
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Figure 5 
Virtual Laboratory on E-module 
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Virtual experimental methods need to be supported by using appropriate media. Physics and Education 
Technology (PhET) contains visualizations of real laboratory components that can be explored in real-time, con- 
cretely, and without risk of injury (Dina, 2020). The E-module developed is equipped with virtual experiments in 
each learning activity using PhET to significantly improve students’ thinking skills, and there are differences in 
learning independence (Kartimi et al., 2022; Ramadan et al., 2020). Through virtual experiments, students can not 
only observe physical phenomena visually but also improve skills, spark questions, critical thinking, and further 
research. By developing E-modules, students can improve their ability to think analytically and critically about the 
learning materials they are studying. These results indicate that PhET simulations in modeling and implementa- 
tion classes are effectively used in terms of improving students’ CT skills toward minimum competency standards 


Discussion 


Based on the research results and explanations from the research results and e-module development process 
carried out, the following results were obtained: 


Module characteristics 


a. This research develops an E-Module using the Problem-Based Contextual Learning model with STEM 

The development of the E-Module in this research used a 4D model to improve students’ low CT skills because 
these skills are very important for 21st-century learning, so they need to be improved in chemistry learning, especially 
thermochemical material. Media development was carried out by following current technological developments. 
Currently, we have entered the digital era, so in the world of education, it is necessary to adapt to advances and 
developments in technology, where technology has the potential to create new learning spaces that are needed 
to facilitate the learning process in the classroom. According to research by Faisal et al., (2020), in the current era, 
students find it easier to learn by utilizing technology and are more interested in digital teaching materials, which 
can later be accessed easily by students using electronic devices such as smartphones. 
b. Teaching modules can be used online via smartphone and laptop 

The learning media developed is media that can be used for theory and practice. Planning and designing 
learning media that can be used as a tool for practical activities in the learning process with this E-module facility 
is more practical. The E-module developed can provide facilities for students to study independently. Students 
can access e-modules anywhere and anytime using a smartphone (Basuki, 2022). The use of information computer 
and technology (ICT)in the learning process can reform education in all aspects (Ishaq et al., 2020), is capable of 
offering an energetic environment for teaching and learning facilities (Shatri, 2020), and can replace the traditional 
learning process with more modern tools and facilities(Motamedi, 2019; Singh, 2016). 
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c. The module developed is made in electronic form based on the Google site. 

Google Sites is an application for creating website-based E-modules. Google Sites is an application that can 
be used to create the most interesting web-based media. The Google site itself was created in March 2008. Users 
can easily, swiftly, and simply design website displays with the help of the Google site. Google Sites can support 
learning by using several available features such as Google Docs, sheets, forms, calendars, awesome tables, etc. 
The Google Site has advantages, such as it is easy to operate and simple to maintain. The application of the site 
does not require a complicated high-level programming language because it is very simple (Riza et al., 2022). Apart 
from that, another advantage is that the Google Site is free and easy to use. Besides, it provides a large capacity 
of 100 MB online storage with sites stored securely on Google. The Google site has also been integrated with 
Google-based applications such as Google Meet, Google Classroom, Google Form, Google Spreadsheet, Google 
Docs, Google Slides, and others (Bhagaskara et al., 2020). Another advantage that the Google site has is that it can 
be accessed by Windows, Linux, Android, etc. Google Sites media was developed to help students develop the 
ability to master concepts and think critically through direct experience because they not only listen to explana- 
tions from teachers but students also carry out other activities such as observing videos and pictures, and carrying 
out simulations (Ciung et al., 2022). 

d. The module developed is equipped with virtual trials or experiments 

The e-Module developed is equipped with virtual experiments using PhET Colorado which can be carried out 
easily online, can save time and money, and can reduce the risk of injury. By using PhET Colorado, students can 
carry out virtual experimental simulations, which can be used to obtain exact data (formula data, not real data) and 
have the advantage of easy operation (Defianti et al., 2021). The use of PhET Colorado cannot completely replace 
real practicum because by using virtual experiments, students conduct and carry out experiments without see- 
ing them directly. Aspects of measurement skills and attitudes cannot be observed directly, so they cannot apply 
data uncertainty due to the data obtained are exact data with the same results. Therefore, it is necessary to use 
approaches, class situations, concepts, and objectives that students must master. 

e. The module supports student activities in achieving indicators of students’ critical thinking skills. 

Students’ CT skills can be seen from the pretest results and initial observation results of students from the 
three schools (high category schools, medium category schools, and high category schools) representing Surakarta 
City, Central Java, Indonesia. Based on the pretest results in Table 8, it can be seen that the average of students 
at three schools representing Surakarta City, Central Java, Indonesia is in the non-critical category. The E-Module 
developed contains activities that can support student activities that contain indicators of students’ critical think- 
ing to be achieved. CT skills are not inherent in students since birth, so they need to be trained in the learning 
process to improve these skills. 


Module eligibility 


In testing the feasibility of the E-Module which was developed, validation by material, language, and media 
experts was conducted and resulted in very good criteria so that the module was suitable for use. Then the teacher 
was assessed from three aspects, namely material, media, and language, showing very good criteria so that the 
E-Module is suitable for use. Apart from that, for feasibility, try-out was also carried out in small groups on five 
aspects, namely media, material, language, learning, and use, showing that the E-Module met the criteria very well 
so that the module was suitable for use. Furthermore, large group tryouts were carried out on five aspects, namely 
media, language, materials, language, learning, and use of modules, showing that the results were suitable for use. 


Module effectiveness 


Students’ CT skills from the three schools were tested after the developed module was applied to learning. 
The results showed an improvement compared to learning before using the E-module. An independent t-test was 
carried out to determine the increase in CT skills after implementing STEM-based E-Module contextual learning 
on chemistry material. The results obtained were a t-test of 0.025 where the significance showed results that were 
smaller than 0.05. meaning that Ho was rejected. Thus, students’ CT skills when using the E-Module are higher 
when compared to students who used conventional media. 

Analysis of aspects of students’ critical thinking skills 


1. Interpretation 
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In the interpretation aspect, schools in the medium category showed an average of 81.5% which was in the 
very critical category, schools in the low category had an average of 80.58% which was in the critical category, and 
schools in the high category had a score of 79.42% with the critical category, so this showed that in identifying 
observations and categorizing phenomena, students can explain phenomena based on previous experience and 
students can identify problems from the phenomena given in each learning activity on thermochemical material. 
This was in line with research conducted by Pertiwi (2018) revealing that students who had interpretation skills or the 
ability to understand the problems they faced well could directly be proportional to students’ critical thinking skills. 


2. Analysis 

In the analytical aspect, schools in the medium category and schools in the low category had an average of 
78.67% in the critical category. Schools in the high category had an average of 80.89% in the critical category, and 
schools with a total of seven indicators had an average of 80.14% in the critical category. This means that students 
can find creative and innovative ideas and find solutions to solve problems presented in learning, detecting argu- 
ments or ideas. Apart from that, it also shows that students can design problem formulations through observation 
and have the ability to design temporary answers to problems given during the learning process in class. Apart 
from that, students can also solve problems in the LKPD by answering the questions in the LKPD. Besides, students 
can conclude the relationship between the results they obtain by answering the questions given in the LKPD by 
analyzing the results from the investigations. This is in line with research conducted by Pertiwi (2018) revealing 
factors that can later influence the analyzing aspect, including students’ ability to answer questions given on 
the LKS according to the concept and given examples of contextual use of technology. In addition, students can 
also introduce the concept and provide related materials. Identifying concepts can improve students’ CT skills. In 
believing something or before doing something, students must carry out a rational process (Listya et al., 2019). 


3. Evaluation 

In the evaluation aspect, there are two indicators, schools in the high category show an average of 79.12%, 
meaning that they are in the critical category. Schools in the low category have an average result of 78.63%, mean- 
ing that they are in the critical category. Schools in the high category have an average of 80.37% included in the 
critical category. The scores obtained by the three schools show that students can consider the arguments used 
in solving a problem related to the results of the analysis, and students can use the right strategy to complete the 
problem formulation completely and correctly related to the results of the analysis. In line with research conducted 
by Benyamin et al., (2021) regarding accuracy in solving a problem, efforts to improve students’ CT skills by using 
STEM-based E-modules in classroom learning activities can improve aspects of student evaluation. The E-module 
developed based on problem-based STEM learning can be accessed using a smartphone as an effort to improve 
students’ CT skills in aspects of student evaluation. During presentations, students learn how to formulate ideas 
and provide explanations for them. During presentations in groups, students are required to be able to deepen 
and develop innovations and ideas in unique ways (Kartimi et al., 2021; Yustina et al., 2022) 


4. Inference 

In the inference aspect, where there are three indicators tested on three schools, schools in the medium 
category get an average of 78.16%, which indicates the critical category. Schools in the high category get an av- 
erage of 79.83%, which indicates that the school is included in the critical category. Schools in the low category 
obtained an average score of 77.33%, which shows that students’ CT skills are in the critical category. The average 
results in the three schools show that students have skills in drawing alternative conclusions, concluding correctly, 
and providing explanations accompanied by supporting evidence during the learning process. This is shown by 
the KKTP that students have passed so that learning using media can run well seen from the average obtained by 
students. The inference prioritizes students’ ability to make inferences from the information acquired and guides 
students to think critically(Pritananda, 2017). 
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Figure 6 
Picture of The Stages of Investigative Activities in The Third Activity (A) Sketch of A Simple Calorimeter Tool Designed by Students 
and Its Dimensions (B) Results of The Calorimeter Tool Made by Students in Groups According to The Sketch Made 
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Based on Figure 4, which is one of the third learning activities at the investigation stage, students carry out a 
bomb calorimeter simulation using a virtual laboratory, then students sketch a simple calorimeter from used materi- 
als found around them. After that, they design a simple calorimeter according to the plan or sketch that has been 
made. The results obtained using a simple calorimeter tool that has been designed are then compared with the 
data obtained using a standard calorimeter and then analyzed. At this stage, the STEM component that emerges is 
science students who can gather information by doing practical work, studying literature, or opening websites on 
the internet. The hope is that students can do this and ultimately find answers to the problems they have formulated 
and be able to determine whether the hypothesis formulated is true or not. Students carry out experiments using 
the PhET Colorado application based on the work steps on the LKPD. In the mathematics component, students can 
use their CT skills to analyze the experimental results obtained and then connect between variables mathematically. 
In the engineering component, students carry out analysis and design experiments and develop work procedures 
for conducting experiments as a way to solve a given problem. The critical thinking ability that is developed is the 
ability to evaluate using problem-solving strategies. Students analyze procedures that have been designed and carry 
out experiments. After that, students carry out data processing and then interpret the data results. 


Figure 7 
The Relationship Between Problem-Based Contextual Learning and STEM Learning Models to Improve Critical Thinking Skills. 
The Red Color Shows Critical Thinking Skills, While the Green Color Indicates a STEM Approach. 
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The research combines STEM-Problem-Based Contextual Learning with critical thinking skills, and the rela- 
tionship between the two is as follows. 


pearing as Critical Thinking Skills Learning Activities 
Syntax components 
Problem Orienta- Science Interpretation (understanding This activity is carried out so that students have an idea of what will be done 
tion and identifying problems) and what must be prepared to achieve the learning objectives and have confi- 
dence that they can complete and achieve the expected learning objectives. 
Organizing Analysis (Identifying relation- | The stage of organizing students aims to ensure that learning activities can 
students ships between statements, run smoothly and under control. Educators guide students to form study 
questions and concepts by groups. At this stage, problems related to the real world are also given ac- 
creating models) cording to the material to be studied. 
Observation Science Evaluation (Using problem To make observations at the observation stage. The observation process is 
solving strategies) carried out based on the problem discourse or after students see the display 
in the form of a power point or video that has been provided. Based on these 
observations, students are expected to be able to identify existing problems 
and formulate them in the form of questions 
Hypothesis Science Evaluation (Using problem Hypothesis stage, at this stage, students are expected to be able to estimate 
solving strategies) answers or temporary solutions from the problem formulation they have 
made, through observations they have made and relate them to the knowl- 
edge they have previously 
Investigation Mathematics Evaluation (Using problem Next, students will search for the truth of their temporary answers at the 
Science solving strategies) investigation stage. Students can do practicums, study literature, or open 
Technology websites on the internet. It is hoped that students will ultimately be able to 
Engineering find answers to the problems they have formulated and be able to determine 
whether the hypothesis they have formulated is true or not. 
Presentation of | Science Explanation (states the Students are asked to communicate the results of their findings so that they 
results results and basic arguments) can be known by other groups 
Conclusion and — Science Inference (making conclu- Next, in the final stage, namely the conclusion, educators and students to- 
evaluation Mathematics sions correctly) gether draw conclusions from the results of the presentations that have been 


made by each group and evaluate the learning process that has been carried 
out so that the next learning activity will be better. 


In the STEM approach, there are four important components, including science, technology, engineering, and 


mathematics (Decoito, 2016; Falloon et al., 2020; Firman, 2016). The STEM approach can be applied to classroom 
learning in various forms of activities, including as models, teaching materials, or Student Worksheets (LKPD) (Cruz 
et al., 2021; English, 2017; Lestari et al., 2018). The application of transdisciplinary STEM learning methods can create 
learning in schools with a collaborative atmosphere that respects sacrifice, initiative, and responsibility and can 
contribute to increasing conceptual understanding, student motivation, and inspiration for students’ critical think- 
ing from students (Shidiq, 2020; Shidiq et al., 2022; Susanti, E., & Fatmawati, 2023)during the COVID-19 pandemic, 
students and teachers could not access laboratories. Thus, innovative pedagogical approaches are required to meet 
these challenges. The current study, therefore, examines the use of contemporary hybrid laboratory pedagogy 
to construct a simple spectrophotometer by implementing Science, Technology, Engineering, and Mathematics 
(STEM. Activities with a STEM approach designed for learning through less study can actively increase students’ 
understanding of the learning material, make learning meaningful, and create a happy atmosphere throughout 
the process of learning (Martynenko et al.,2023; Nursyahidah et al., 2023). By implementing learning with inte- 
grated digital media, STEM in learning has the advantage of developing elements of the conceptual knowledge 
of the students and can improve students’ literacy skills effectively (Asrizal et al, 2023). So, the improvement of 
class XI students’ CT skills through the implementation of the STEM approach in digital learning materials, such 
as the STEM problem-based contextual learning approach in website-based e-module learning materials, is very 
effective on thermochemical materials. 
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Conclusions and Implications 


In this research, digital learning media is developed in the form of an E-Module STEM-Problem Based Contex- 
tual Learning, displayed electronically on the Google site application. The E-module developed is equipped with 
integrated experiments with the PhET Colorado application. Learning media is designed to fulfill the four STEM 
components in each learning syntax. The advantage of the E-Module STEM-Problem Based Contextual Learning 
package on the website is that it helps students understand chemistry learning materials more easily. Based on 
the feasibility test carried out in the research, the module is suitable for use for learning thermodynamics in terms 
of the material, media, and language aspects. Feasibility try-out with an average of small-scale try-out results in 
a score of 84.91% in the category of very good and usable. Large-scale try-outs yield an average score of 84.06% 
in the very good category, making it appropriate for educational use. With an average N-Gain value of 0.70 with 
strong criteria, the effectiveness test indicates that the module has been successful in enhancing students’ critical 
thinking skills in thermochemistry courses. The use of chemistry E-module on thermochemical materials ranging 
from low to high improved critical thinking skills, based on the research and development findings. Therefore, 
educators can utilize this E-module as an alternative when selecting instructional materials to help students de- 
velop their critical thinking skills. 
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Abstract. A central objective of science 
education is to foster a profound 
comprehension of fundamental scientific 
principles among students. Research has 
shown that a highly integrated knowledge 
structure is a key factor in achieving a deep 
understanding. This research has developed 
a friction force conceptual framework to 
model students’ different understandings 
of friction force from a knowledge 
integration perspective. Utilizing the 
established conceptual framework, 

this study devised and implemented 

an evaluation of friction force among a 
cohort of 598 grade-10 students in China. 
The assessment outcomes were then 
analyzed quantitatively and qualitatively. 
The results show that the conceptual 
framework model effectively represents 
the knowledge structures of students at 
different levels of knowledge integration, 
and the students’ understanding of 
friction force can be divided into three 
levels: novice, intermediate, and expert. 
Furthermore, emphasizing the passivity 
of friction force can help students gain 

a deeper understanding of the concept 

of friction force, thereby forming a more 
comprehensive knowledge structure. 

This study provides an effective strategy 
for students to progress from novices to 
experts. 
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Introduction 


Scientific education should foster the development of transferable 
knowledge and skills, enabling students to meet the demands of modern 
production and daily life (National Research Council, 2012). However, exam- 
oriented education often prioritizes the memorization of knowledge over 
its comprehension. Furthermore, problem-solving is frequently reduced 
to recalling superficial features of problem contexts (Alonso, 1992; Bao & 
Koenig, 2019). This approach leads to students excelling in solving familiar 
problems but struggling when faced with novel or unprecedented scenarios 
(Chiu et al., 2007; Kim & Pak, 2002; King, 1992; Nurrenbern & Pickering, 1987), 
indicating a superficial understanding of concepts and a lack of profound 
comprehension (Champagne et al., 1982; Krathwohl, 2002; Nakhleh, 1993; 
Rivet & Krajcik, 2008). Numerous studies analyzing students’ problem-solving 
behaviors have identified the organization of knowledge as a pivotal differ- 
entiator between expert and novice learners (Champagne et al., 1982; Chi 
et al., 1981; Hardiman et al., 1989; Sabella & Redish, 2007; Snyder, 2000). The 
variance in students’ conceptual understanding can be modeled through 
the construction, activation, and linking of concepts (Linn, 2006; Liu et al., 
2011; Sabella & Redish, 2007). 

To enhance students’ conceptual understanding and to facilitate tar- 
geted instructional guidance, it is essential to assess the level of students’ 
knowledge integration. In recent studies, Bao et al. (2019) have developed 
a conceptual framework tool to model the cognitive structures and the rea- 
soning pathways during problem-solving of students as they progress from 
novices to experts. The framework delineates three distinct developmental 
levels of knowledge integration: novice, intermediate, and expert. Novices’ 
knowledge structures display fragmented features with loosely connected 
concepts, closely tied to the contexts encountered in textbooks and lectures. 
Throughout the problem-solving process, novices typically employ strategies 
that match contextual features with memorized operational procedures and 
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equations (Chen et al., 2020; Chi et al., 1981; Larkin et al., 1980). Students at an intermediate level progressively 
cultivate knowledge structures that are partially integrated, which facilitates their capacity to perform logical 
analyses within contexts that are familiar to them. However, the incomplete integration of knowledge limits their 
ability to apply existing knowledge to unfamiliar contexts (Bao et al., 2002; Bao & Redish, 2006; Zou et al., 2023). 
In contrast, experts possess highly integrated knowledge structures centered around the central idea, facilitating 
the flexible application of knowledge to solve problems across various contexts (Chi et al., 1981). The conceptual 
framework for student learning is tailored to specific content areas, thus requiring the creation of distinct frame- 
works for individual conceptual subjects. This model has been successfully developed and implemented across 
various physical science topics, including, for example, force and motion (Nie et al., 2019), momentum (Xu et al., 
2020), Newton's third law (Bao & Fritchman, 2021). 

Friction force is a fundamental concept in physics, and integral to the curriculum from elementary to university 
education. Despite its significance, research in this area remains limited, primarily focusing on students’ misconcep- 
tions and the challenges they encounter when solving problems (Arons, 1997; Besson et al., 2007; Caldas & Saltiel, 
1995; Kuzulcuk et al., 2021). However, the specific causes of these difficulties remain unknown. Furthermore, this 
dearth of research on varying levels of comprehension of the friction force impedes the development of tailored 
pedagogical approaches that could effectively address these educational challenges. To assess students’ comprehen- 
sion of friction force and elucidate the challenges they encounter in understanding this concept, this study aimed 
to model students’ different conceptual understandings of friction force from the lens of knowledge integration. 
Specifically, the following work was done in this study: 

(1) Construct a conceptual framework for friction force to simulate the knowledge structures of students 
with varying levels of conceptual understanding. 

(2) Design an evaluative instrument grounded in the conceptual framework to investigate students’ com- 
prehension of friction force and infer their knowledge structures. 


Conceptual Framework 
Friction Force Conceptual Framework 


A conceptual framework is a tool used to describe the knowledge structures of students on a specific topic. The 
elements within a conceptual framework include not only concepts but also the basic characteristics of problem 
situations and variables, as well as task goals. The interconnections among the elements reveal the knowledge 
structures of students with varying levels of conceptual understanding and illuminate their reasoning processes 
as they apply these elements to problem-solving. This research has formulated a conceptual framework for friction 
force, acting as a theoretical construct to outline the knowledge structure of students. The framework’s components 
and structure were derived and refined through expert analysis of the physical concepts documented in literature 
and the learning behaviors of students. Friction force encompasses various types, and the focus of this study was on 
static friction and sliding friction, which are topics covered in the physics curriculum at the high school level in China. 

Drawing on prior research, the initial step in constructing a conceptual framework involves identifying a 
central concept that acts as an anchor, integrating the related concepts (Bao & Fritchman, 2021; Dai et al., 2019; 
Liu et al., 2022; Nie et al., 2019; Tong et al., 2023; Xie et al., 2021; Xu et al., 2020; Zou et al., 2023). Friction force, a 
“passive” force (Arons, 1997), exhibits an inherently complex phenomenology that is contingent upon the materi- 
als in question and the conditions under which they are applied (Arons, 1997; Besson et al., 2007). This passivity is 
manifested in three key aspects: 

(1) Conditions for the generation of friction force: firstly, mutual contact and pressing between objects; 
secondly, the contact surface is rough; and thirdly, the presence of either relative motion or a tendency 
for it between the objects. 

(2) Direction: The direction of friction force is always opposite to the direction of relative motion or relative 
motion tendency of the object. The objects here refer to those that can be considered as mass points. 
When dealing with complex objects, such as the direction of friction force on a car, it is necessary to 
separately discuss the motion situation of the front and rear wheels relative to the ground. 

(3) Magnitude: The magnitude of friction force is related to the properties of the two contacting surfaces 
(such as their roughness) and the normal force between them. 

According to the passivity of friction force, whether it exists or not, as well as its magnitude and direction, 
is determined by other external forces and the object’s state of motion (e.g. at rest, uniform straight-line motion, 
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and accelerated motion). When addressing problems, experts take into account the passivity of friction force and 
analyze its actual effect based on the object's actual state of motion and the forces acting upon it. This contrasts 
with novice students, who often simplistically regard friction force as a resistive force (Caldas & Saltiel, 1995; Clem- 
ent, 1993), and directly apply the equation 52 = MF, to determine its magnitude for any and every friction force 
whether slipping is about to occur or not. (Arons, 1997; Boudreaux & Elby, 2020; Clement, 1993). In light of the 
preceding analysis, a profound understanding of the passivity of friction force is an important criterion to distin- 
guish between experts and novices in dealing with problems related to friction force. Therefore, in this research, 
the passivity of friction force is chosen as the central idea. 


Figure 1 
Friction Force Conceptual Framework 


The passivity of friction force 
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Contextual Features& Variables 


After thorough analysis and expert deliberation, the conceptual framework for friction force has been crafted 
andi is illustrated in Figure 1. The central idea of this framework is the passivity of friction force, which integrates other 
related concepts within the subject matter. As previously stated, the inherent passivity of friction force necessitates 
that its effect within a given context is inferred from the actual state of the object’s motion. Given other conditions 
are met (such as the contact surface is rough), the occurrence of relative motion between objects readily implies 
the presence of sliding friction. Conversely, when the object is relatively stationary, it is not possible to determine 
whether there is a tendency to relative motion directly by observing the object's motion state. So, it is impossible 
to determine whether there is static friction. Newton’s laws of motion are the key link between force and motion, 
through Newton's laws of motion, the force on an object can be evaluated in terms of its motion (Arons, 1997; Ha 
et al., 2013). Hence, Newton's laws of motion are essential in analyzing friction force. Accordingly, the second tier 
of the conceptual framework encompasses Newton's laws of motion. The Newton's laws of motion discussed here 
encompass both predictive and interpretive methodologies for analyzing problems using these principles (Ha et al., 
2013). The predictive approach allows for the forecasting of an object's subsequent motion when the quantitative 
characteristics of the forces acting upon it are known, such as their magnitude and direction. On the other hand, 
the interpretive method involves deducing the properties of the forces by observing the motion characteristics of 
the object when the forces’ quantitative attributes are not precisely defined. Proficiency in Newton's laws of motion 
enhances the comprehension of friction force's passivity and aids in problem-solving (Ha et al., 2013). 
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The central idea is also interconnected with other concepts, such as the calculation equation for kinetic fric- 
tion, reference of frame, kinematics, and force synthesis, which may collectively contribute to addressing problems 
related to friction force. The third tier involves intermediate reasoning processes and operational procedures, 
including force analysis, and mathematical and logical manipulation processes. For expert students, the third tier 
effectively connects with the concepts in the first and second tiers. However, novice students often apply these 
procedures as memorized routines for problem-solving, creating a disconnect from the central idea. The bottom 
tier comprises contextual elements, which commonly pertain to the problem's design attributes. These can be 
modified to generate a variety of contexts and task environments. For most problems related to friction force, 
the task goals typically involve several types: whether friction force is present, identifying its type, quantifying its 
magnitude, and determining its direction. 

Furthermore, the arrows linking disparate components signify possible reasoning pathways for individuals 
ranging from novices to experts. Solid arrows depict the reasoning paths of experts, while dashed arrows repre- 
sent possible paths for novices. Experts, utilizing the central idea, establish comprehensive connections among 
related concepts, reasoning processes, operational procedures, and contextual features within their knowledge 
structure. However, novices tend to combine contextual features with memorized operational procedures to 
constitute a disjointed knowledge structure. As illustrated in Figure 1, the linkages are marked with bidirectional 
arrows, signifying that an interaction can originate from either terminus. However, some students may possess 
unidirectional links between some connections, reflecting a fragmented knowledge structure. In summary, the 
conceptual framework integrates these levels and task objectives to visually represent the knowledge structures 
of students with varying levels of conceptual understanding. 


Modeling Knowledge Structures of Students at Different Levels 


Employing the developed conceptual framework, this study stratifies students’ comprehension of friction 
force across three levels: novice, intermediate, and expert: 

(1) Novice level: At this level, students exhibit fragmented knowledge structures, and establish only local- 
ized connections between contextual features of problems and memorized operational procedures. 
For instance, students can accurately calculate the magnitude of the friction force acting on an object 
moving on a horizontal surface using the formula f = fil, = mg . Yet, when dealing with an 
object on an inclined plane, they still apply the formula ia = Lumg to calculate (Besson et al., 2007). 
Furthermore, students misinterpret the effect of friction force (Caldas & Saltiel, 1995), which impedes 
relative motion, incorrectly as “hindering motion’. This leads to errors in determining the direction of 
friction force, with students believing “the direction of friction force is opposite to the direction of object 
motion” (Prasitpong & Chitaree, 2010) or “the direction of friction force is opposite to the direction of 
external force” (Prasitpong & Chitaree, 2010). Students at this level lack a fundamental understanding 
of the passivity of friction force. Their comprehension of friction force is largely derived from everyday 
life experiences, and their reasoning pathway can be described as “contextual features + memorized 
equations — task goals”. 

(2) Intermediate level: At this level, students possess a fundamental understanding of the passivity of 
friction force and acknowledge that friction force is not inherently a resistive force but can serve as a 
driving force contingent on the context. Consequently, when addressing problems, they evaluate the 
presence and effect of friction force based on the specific situation. These students have developed a 
more complex and interrelated knowledge structure, however, their lack of familiarity with the applica- 
tion of the Newtonian mechanical interpretive method hinders the complete integration of Newton's 
laws of motion within this framework. Therefore, their analysis of object motion is limited to determin- 
ing the presence of relative motion. In situations where relative motion is not evident, they may face 
challenges in evaluating the likelihood of relative motion tendency between objects (Boudreaux & 
Elby, 2020). Furthermore, it is noteworthy that these students also tend to depend on memorization 
strategies, thus they excel in addressing problems within familiar settings but face challenges in rea- 
soning and analysis when encountering unfamiliar contexts (Bao et al., 2002; Bao & Redish, 2006). For 
example, while proficient in analyzing the friction force acting on a pushed box, students demonstrate 
uncertainty when confronted with the analysis of the friction force experienced by an individual on a 
sled during skiing (Prasitpong & Chitaree, 2010). Therefore, their reasoning pathway can be described 
as “contextual features — the passivity of friction force — task goals”. 
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(3) Expert level: At this level, students possess a profound understanding of the passivity of friction force. 
Utilizing this as an anchor point, they integrate Newton's laws of motion with other related concepts 
and form a comprehensive and coherent knowledge structure. In tackling friction-related problems, 
these students first thoroughly observe the phenomenon. Then, they apply Newton's laws of motion 
to analyze the precise effects of friction, avoiding reliance on rote memorization (Ha et al., 2013). 

Figure 2 illustrates an example problem, and the following discussion demonstrates the reasoning process 
of the expert based on this example. Experts possess a profound understanding of the passive nature of friction 
force and recognize that the friction acting on an object is contingent upon its state of motion as well as other 
active forces. Evidently, the pull is the active force. Consequently, their initial step is to analyze the motion state of 
the object. Due to the uncertainty regarding whether relative motion has occurred between objects A and B, it is 
not possible to ascertain whether the friction force is sliding or static. However, Newton's third law tells them that 
the friction force between objects A and B always occurs in pairs,and /,, =—/,,, . Therefore, considering objects 
A and Bas a system is a good approach to solving the problem. By applying Newton's second law to the system, 
they calculate the acceleration (the predictive methodology of Newton's laws). Following this, they proceed to 
analyze each object individually, again utilizing Newton's second law to determine the magnitude of the friction 
force (the interpretive methodology of Newton’s laws). Guided by the central idea of the passivity of friction force, 
experts will not fall into the trap of directly applying the formula for sliding friction. 

Their reasoning pathways can be characterized as “contextual features — passivity of friction force + Newton's 
laws of motion — task goals”. 


Figure 2 
Sample Question 
Question 


As shown in the figure, m, =2kg , m, =lkg , Aand B have a kinetic friction coefficient of 0.5, 


and the horizontal surface is smooth. When a 10 N horizontal force F is applied to pull A, what is 
the friction force between A and B? 
(A)ON 
(B)5N 


(C) 15N 


10 
D) —N 
’ 3 


2 


(E) On 
3 


In summary, the conceptual framework delineates the knowledge structures of students across different 
levels of conceptual understanding, thereby explaining the challenges they face in understanding friction force. 
Furthermore, supplementary interviews are implemented to delve deeper into the students’ reasoning pathways, 
thereby verifying the plausibility of the conceptual framework. 


Research Methodology 
Assessment Design 


Drawing upon the conceptual framework for friction force, an evaluation instrument consisting of 14 multiple- 
choice items has been developed, designed to assess the level of students’ knowledge integration. Some questions 
are adapted from previous research (Besson et al., 2007; Prasitpong & Chitaree, 2010), while the remainder are 
designed based on college entrance examination questions and practice problems. 

Studies suggest that conceptual-framework-based assessments are particularly effective for evaluating both 
the integration of knowledge and the profundity of comprehension (Bao & Fritchman, 2021; Dai et al., 2019; Nie 


77/1 


https://doi.org/10.33225/jbse/24.23.767 


Journal of Baltic Science Education, Vol. 23, No. 4, 2024 


ASSESSMENT OF STUDENT KNOWLEDGE INTEGRATION IN LEARNING FRICTION FORCE ISSN 1648-3898 | /printy 


Ee ISSN 2538-7138 /oniines 


et al., 2019; Zou et al., 2023). The assessments effectively leverage specific question design elements—such as 
contextual saliency and the connectedness of knowledge—that signify a level of knowledge integration. 

Regarding the contextual saliency in question design, the assessment utilizes both typical and atypical ques- 
tions, a strategy that has proven effective for developing assessments based on the conceptual framework (Bao & 
Fritchman, 2021; Dai et al., 2019; Zou et al., 2023). The contexts presented in typical questions are familiar to stu- 
dents and can be addressed using memorized solutions. In contrast, the context in atypical problems is unfamiliar 
to students, hence they need to synthesize the knowledge they have learned and make reasonable inferences to 
solve the problems. Novice and intermediate level students are capable of resolving problems within typical con- 
texts utilizing memory-based strategies; however, they encounter difficulties when faced with atypical contexts. 
Conversely, experts demonstrate the ability to address problems across a wide range of contexts. 

In terms of the connectedness of knowledge, this study draws upon previous theories (Biggs & Collis, 1982; 
Linn, 2006), and appropriately modifies them in accordance with the conceptual framework. The initial link types 
have been streamlined into three distinct levels: single-link, multi-link, and integrated-link. In single-link ques- 
tions, students are tasked with establishing a singular association between discrete contextual components and 
concepts, which are typically solvable by recalling the relevant methodologies and operations. Multi-link problems 
require the establishment of associations among a broader range of contextual elements, multiple conceptual 
components, and operational steps; yet these associations are typically constrained in scope. Conversely, solving 
integrated-link problems requires a cohesive knowledge structure. 

Expanding upon current methods for assessing knowledge integration, the test on friction force consists of 
14 questions that are divided into three categories. Table 1 provides an overview of the types of questions and 
the expected problem-solving abilities of students at various stages of knowledge integration. The score patterns 
are denoted using the notation L/M/H, which corresponds to the levels of low, medium, and high performance, 
respectively. The scores are sequentially arranged to represent the progression from novice to intermediate and 
finally to expert proficiency levels. A specific sequence, such as LMH, is indicative of low scores attributed to novice 
participants, medium scores for those at the intermediate level, and consistently high scores for individuals at the 
expert proficiency level. 


Table 1 
Questions Type and Score Patterns 


Question Set Question Type Score Patterns 
Set 1 (Q1, Q5, Q6, Q10, Q13) Single-link typical LHH 
Set 2 (Q2, Q4, Q7, Q8) Multi-link typical LMH 
Set 3 (Q3, Q9, Q11, Q12, Q14) Integrated-link atypical LLH 


Assessment Properties of the Test 


As shown in Table 2, the difficulty index is determined to be .575, and the discrimination index is calculated 
to be .527. According to DeVellis (1991), these values fall within an acceptable range. The reliability of the assess- 
ment instrument is evaluated using Cronbach's @ coefficient. The derived value of .77 exceeds the threshold of 
.65, indicating a reliable level of consistency within the test’s measurements (DeVellis, 1991). 


Table 2 
Item Difficulty and Item Discrimination from the CTT Analysis 


Question Difficulty Discrimination 
Ql 724 444 
Q2 595 642 
Q3 A18 487 
Q4 605 588 
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Question Difficulty Discrimination 

Q5 116 468 

Q6 609 463 

Q7 503 550 

Q8 670 568 

Qg 568 530 

Q10 686 460 

Qi1 A441 133 

Q12 488 Al 

Q13 682 485 

Q14 347 486 

Average 575 527 


The validity of the test was evaluated from both content validity and construct validity. In the development of 
the assessment instrument, in addition to the author, several high school physics teachers, educational researchers, 
and three graduate students specializing in physics within the team were collaboratively involved in the compila- 
tion process. By encompassing multiple pilot tests and feedback from faculty and graduate students at the author's 
research institution, the design underwent a meticulous development and refinement process. The final version 
of the test content was unanimously deemed scientifically accurate and effective by the design and evaluation 
team; therefore, the test has good content validity. 

Exploratory factor analysis (EFA) was performed to evaluate the construct validity of the test. As shown by 
the scree plot in Figure 3, there are three factors with eigenvalues greater than one, and they can explain 59% 
of the variance (with factors 1, 2, and 3 accounting for 23%, 19%, and 17%, respectively). Therefore, the student's 
performance on this test can be fully explained by three factors. Table 3 presents the factor loadings for each item, 
where a factor loading represents the degree of correlation between an item and its underlying factor (loadings 
below 0.35 have been omitted). It is evident that items Q1, Q5, Q6, Q10, and Q13 exhibit substantial and significant 
associations with Factor 1. Items Q2, Q4, Q7, and Q8 are closely related to Factor 2, while items Q3, Q9, Q11, Q12, 
and Q14 are closely associated with Factor 3. This distribution is entirely consistent with the pre-established subject 
categories, thereby indicating that the test possesses strong construct validity. 


Figure 3 
Scree Plot of EFA 


Eigenvalue 


Factors 
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Table 3 
Pattern Matrix 


Item Factor 1 Factor 2 Factor 3 


Q13 80 


Qi 16 


Q10 16 


Q6 is) 


Q 35 


Q4 94 


Q7 89 


Q2 83 


Q8 38 
Qi 95 


Q12 81 


Q9 81 


Q3 2B 


Q14 68 


Participants 


This study involved 598 grade-10 students from 12 classes in an upper-secondary school in China, compris- 
ing 317 female and 281 male students, with ages ranging from 15 to 16 years old. Among them, four classes were 
designated as“better classes”, where students exhibited superior performance in physics, while the remaining eight 
were considered ordinary classes with average physics achievements. This stratification ensured a diverse range of 
abilities among the participants in the study. All participants had been exposed to the physics curriculum, specifi- 
cally covering topics on friction force and Newton's laws of motion, which is a prerequisite for the study. Prior to the 
study, all participants were briefed on its objectives and the voluntary nature of their involvement. They were assured 
that their participation would not affect their physics grades, and the test results would be used independently.- 


Data Collection and Analysis 


The assessment was conducted in December 2023. The duration of the assessment was 40 minutes. Subsequent 
to the assessment, a voluntary sample of 30 students from the participant pool was selected for interviews. These 
interviews were conducted within one-week post-assessment, with each session lasting approximately 30 minutes. 

After the test, the students’ understanding of the concept of friction was explored and verified by quantita- 
tive and qualitative methods. Previous research has shown that students’ level of knowledge integration can be 
revealed in the score gap of questions with different link types and context saliency (Xie et al., 2021; Xu et al., 
2020; Zou et al., 2023). Therefore, quantitative analysis focuses on identifying differences in students’ performance 
in various problem designs, thereby inferring students’ knowledge structure. The statistical significance of the 
comparative analysis between results derived from different question designs is ascertained through the applica- 
tion of one-way analysis of variance (ANOVA). Subsequent to this, the analysis was deepened by employing the 
paired samples t-test to examine pairwise group differences, complemented by Cohen's d to estimate the effect 
size. The qualitative analysis is further used to analyze the knowledge structures of students with varying levels of 
knowledge integration of friction force. 
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Research Results 
Results of the Quantitative Study 


The scores of students on questions categorized by varying question types are detailed in Table 4. The 
results show that students performed best in single-link typical questions and performed worst in integrated-link 
atypical questions, which aligns very well with our previous expectations. A one-way analysis of variance (ANOVA) 
revealed statistically significant differences among the scores for the three question sets [F(2, 1791) = 186.644, p 
< .001]. The distinctions were more explicitly revealed through the application of paired t-test [t..y(997) = 8.407, p 
< .001, d= .344; t997) = 24,545, p< .001, d= 1.004; Eieoet) = 17.699, p < .001, d=.724]. In summary, the result 
suggests that questions designed with varying link types and context saliency are effective in assessing the level 
of students’ knowledge integration. 


Table 4 
Students’ Scores on Different Question Sets 


Question Set Link type Context type M SE 
Set 1 Single-link Typical 173 010 

Set 2 Multi-link Typical 656 012 

Set 3 Integrated-link Atypical 361 012 
Total Score 591 011 


Notes: SE = Standard Error; M = Mean 


This assessment served two purposes: first, to validate the effectiveness of the friction force conceptual 
framework; and second, to infer students’ levels of knowledge integration by analyzing their performance across 
different question sets. To achieve these objectives, students were initially categorized into five groups based on 
their total test scores, with each group representing 20% of the sample. Subsequently, the study compared the 
scores of each group on different types of questions, as illustrated in Figure 4. 


Figure 4 
Scores on Different Types of Questions by Students of Different Levels (With Error Bars Denoting Standard Error) 


1.0 


Single-link typical 
—® Multi-link typical 
—@ Integrated-link atypical 


0.8 


0.6 


0.4 


Question Set Score 


0.2 
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As shown in Figure 4, students in the lower group (0-20%) perform consistently poorly across all question 
designs. These students can only address a subset of the single-link typical questions, yet they struggle with the 
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majority of multi-link typical and integrated-link atypical questions. Consequently, this suggests that they have 
not grasped the fundamental concept and remain at an early stage in the process of knowledge integration. 
Intermediate students (20%-80%) are able to solve most single-link typical and multi-link typical questions but 
perform poorly on integrated-link atypical questions. This implies that these students have acquired a rudimen- 
tary comprehension of the concept of friction force, yet they have not fully developed an integrated knowledge 
structure. The findings indicate that these students have attained a moderate degree of knowledge integration. 
They excel at employing memorization strategies to address questions in familiar contexts but struggle with the 
flexible transfer and application of knowledge to unfamiliar contexts. Ultimately, students in the advanced group 
(80%-100%) demonstrate nearly equivalent performance across these three types of questions and achieve rela- 
tively high scores in each category. This indicates that these students have profoundly understood the central idea 
and integrated it with other related concepts to construct a comprehensive knowledge structure. Thereby, they 
have achieved a level of expert. 

Furthermore, by observing the slope change of the curve, it can be seen that the students’ understanding 
of friction force goes through three different stages. The initial stage (0-40%) is defined by the aggregation of 
information. At this stage, students gather a variety of contextual variables and associated concepts into their 
knowledge structures. This process is guided by the problem's apparent characteristics. Consequently, their abil- 
ity to solve problems in familiar contexts (including single-link typical questions and multi-link typical questions) 
is growing at almost the same rate, while their ability to tackle atypical problems improves at a slower pace. The 
second stage (40%-80%) involves the construction of connections between concepts. At this stage, students have 
gradually gathered sufficient key information and fundamental concepts, and they begin to establish connections 
between them. As a result, the gap between students’ performance in solving multi-link typical and single-link 
typical problems gradually decreases. The third stage (Top 20%) is the comprehensive integration of the knowledge 
structure. Here, students continuously build connections between concepts and variables that are centered on the 
central concept. This process ultimately leads to the formation of an expert-level understanding of the concept. 
As a result, the gap in the performance of students at this level on all types of questions gradually decreases and 
eventually converges. 

The analysis reveals that the evolution of students’ conceptual understanding aligns with the process of 
knowledge integration. That is, students continuously learn new knowledge and gradually connect it with the 
central idea, eventually forming an integrated knowledge structure. This facilitates the transition from novice to 
expert and achieves a deep understanding of the concept. 


Results of the Qualitative Study 


Within one week of the student taking the test, a think-out-loud interview was conducted with 30 individu- 
als from the test group, including novice (10), intermediate (10), and expert (10) students. During the interviews, 
students were instructed to provide answers to questions along with their reasoning processes. 

Novice level: In the problem-solving process, these students generally associate the context of the problem with 
operational steps and equations they have memorized to derive a solution. Moreover, students at this proficiency 
level frequently exhibit a deficient grasp of the core concept, as illustrated by the subsequent interview excerpts: 


Student A: (Answered B to Q1) “The object in the problem moves to the right. Friction force hinders the motion of an object, 
which means that the direction of friction force is opposite to the motion of the object, so the direction of friction 
force is to the left.” (Answered A to Q4) “Similar to the previous analysis, the object moves to the right, so, the fric- 
tion force is directed to the left.’ 


Student B: (Answered A to Q10) “Because the masses of the three objects are the same, and the friction coefficient with the 
ground is the same, we can determine that the friction force on the three objects is equal using the equation 


f= HEY? 
Student C: Answered A to Q11) “The context described in this question is commonly encountered in daily life. When writing 
with a brush moving to the right, the paper tends to slide to the right. Placing an object on the left side of the paper 


prevents sliding. Therefore, the object exerts friction force on the paper, directed to the left.” 


Q1 represents a quintessential example, one that students have previously encountered within analogous 
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problem scenarios. Thus, recalling the outcomes of previous questions enables them to provide the correct response. 
However, when explaining the reasoning, student A referred to the effect of friction force as resisting motion instead 
of resisting relative motion, indicating an incorrect grasp of the central idea. This misunderstanding is reflected in 
their inability to successfully answer Q4, which is a multi-link question. Similarly, student C, like student A, did not 
correctly understand the essence of friction force. Consequently, when attempting to solve question Q11, which 
involves integrated concepts, student C had to rely on intuition to guess the answer. Furthermore, student B’s 
response reveals that novice students can memorize equations but do not grasp their meaning, leading to a misun- 
derstanding of the normal force F,, as the gravitational force of the object when applying the formula f = £4F,,. 

In summary, novice students’ understanding of friction force is superficial, lacking a thorough comprehension 
of its essence. Most of their perception comes from life experience, for instance, they commonly perceive friction 
force as a resistive force, which always acts opposite to the direction of motion. The interview results corroborate 
Table 4, indicating that novice students can successfully solve some single-link typical questions but struggle with 
multi-link typical and integrated-link atypical questions. 


Intermediate Level: These students have developed an appreciation for the passivity inherent in friction force, 
moving beyond the rudimentary perception of friction as solely a force of resistance. This enhanced understanding 
is exemplified in the forthcoming interview excerpts: 


Student D: (Answered B to Q1 and Q4) “In the first question, the object is moving to the right relative to the ground. Based on 
the effects of friction force, which impedes the relative motion between objects, so the object experiences friction 
force from the ground to its left. In the fourth question, when the object is in segment AC, its velocity is less than 
that of the conveyor belt, so the object moves to the left relative to the conveyor belt, experiencing friction force 
from the conveyor belt to its right.” 


It is evident that intermediate-level students possess a foundational grasp of the concept of frictional pas- 
sivity and are able to solve most single-link and multi-link questions. However, their knowledge structure remains 
incompletely synthesized, which often leads to challenges in addressing problems of integrated link. This issue is 
exemplified by the interview excerpts presented below: 


Student E: (Answered E to Q3) “There is no relative motion between people and objects, so there is no friction force between 
them.” (Follow-up question: What are the conditions for the existence of friction force?) “It is mentioned in the 
textbook, there are three conditions. First, rough contact surfaces; second, compression between objects; third, 
relative motion or relative motion trends between objects.” (Follow-up question: Can we conclude that there is no 
friction force if there is no relative motion between them? Is there a relative motion trend between them?) “Prob- 
ably not, because the person is stationary relative to the escalator, and I'm not quite sure either.” 


Student F: (Answered A to Q6) “Initially, consider the diagram depicted on the left. The foot exerts a force in a posterior direc- 
tion, resulting in a relative retrograde motion trend of the foot against the ground. Consequently, the friction force 
acting on the foot is directed anteriorly. Similarly, as depicted in the diagram on the right, when the foot exerts 
a force in an anterior direction, it exhibits a forward motion trend relative to the ground, and thus, the frictional 
force on the foot is directed posteriorly.” 


Analysis of the above interviews shows that in the absence of relative sliding between objects, intermediate- 
level students rely primarily on intuition or conjecture to assess whether there is static friction between objects, 
rather than applying Newton's laws of motion. Subsequent interviews show that while these students had mas- 
tered Newton’s laws of motion, they had not yet incorporated these principles into their knowledge structure of 
friction force. 


Student G: (Answered A to Q11) “When writing to the right, the brush moves to the right relative to the paper, so the brush is 
subjected to the leftward friction given to it by the paper. And according to Newton's third law, the brush exerts a 
rightward friction force on the paper. But the paper remains stationary, so the paper has a tendency to move to 
the right with respect to the object, so the paper is subjected to the leftward friction exerted on it by the object.” 
(Follow-up question: Does that object experience the friction force of the paper against it?) “According to Newton's 
third law, ifan object has a left-facing friction against the paper, then the paper has a right-facing friction against 
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the object. But in this case, the object has a net force to the right in the horizontal direction, and according to 
Newton's laws of motion, it should move to the right. Oh, | was wrong in my analysis, there is no friction between 
the object and the paper.” 


Newtonian mechanics can be approached through two distinct methodologies: the predictive method and 
the explanatory method. Interviews have revealed that intermediate level students demonstrate proficiency in 
utilizing the predictive method, adeptly deducing the motion states of objects by examining the forces exerted 
upon them. Conversely, the application of the explanatory method presents a more formidable challenge for these 
students. While a thorough comprehension of force dynamics enables accurate predictions of an object's trajectory, 
the analysis of friction force introduces complexities. Friction force, not being a fundamental force governed by 
universal laws, necessitates the complementary use of the explanatory method alongside the predictive approach. 

In summary, intermediate-level students have gained a preliminary understanding of the passivity of friction 
force and recognized the importance of analyzing the effects of friction force based on the motion state of objects. 
However, intermediate-level students still fall short in their comprehension of friction force, due to difficulties in 
integrating Newton's laws of motion into their knowledge structure of friction force. This leads to challenges in 
determining the relative motion tendencies between objects. 

Expert Level: Students at the expert level are capable of solving almost all questions. Particularly, students at 
this level seem to have applied their knowledge of Newton’s laws of motion to analyze questions involving friction 
force, as evident from the interview excerpts below: 


Student H: (Answered D to Q3) “Firstly, analyze the motion state of the person; there is an acceleration directed towards the 
upper right corner. This acceleration can be decomposed into vertical and horizontal components. Consequently, 
the net force in the vertical direction is upward, indicating a state of apparent weight increase. In the horizontal 
direction, the net force is to the right, and since there is no external force acting horizontally, there must be a fric- 
tion force acting to the right. As the person is stationary relative to the escalator, this friction is static friction.” 


Student I: (Answered A to Q9) “Firstly, analyze object B, which definitely experiences friction force from the slope because it 
meets the conditions for friction force to occur. Therefore, the acceleration of object B along the slope is less than 
the component of gravitational acceleration along the slope. Next, analyze object A. Since object A is relatively 
stationary with object B, their velocities are consistent at all times, and thus their accelerations are the same. Suppose 
object A does not experience friction force from object B; then its acceleration along the slope would definitely be 
greater than that of object B, making this assumption unreasonable. Therefore, object A does experience friction 
force from object B, and the direction of this friction is upward along the contact surface.” 


Student J: (Answered C to Q171) “During the process of moving the brush to the right, the brush moves to the right relative 
to the paper. Therefore, the paper exerts leftward friction force on the brush. According to Newton's third law, the 
brush exerts rightward friction on the paper. Since the paper is stationary, the table applies leftward friction force 
on the paper.” (Follow-up question: Why can't it be the object providing leftward friction to the paper?) “Because if 
the object imparts leftward friction to the paper, according to Newton's third law, the paper must exert rightward 
force on the object. According to Newton's first law, the object would change its state of rest and move to the 
right. However, the object remains stationary, which implies from Newton's first law that there is no friction force 
between the object and the paper.” 


The reasoning processes of these three students reveal a deep understanding of the passivity of friction force. 
When addressing problems, they determine the actual effects of friction force by analyzing the motion state of 
objects. In situations where there is no relative motion between objects, they can apply Newton’s laws of motion 
to assess the tendency of relative motion, and thereby ascertain the existence and specific effects of static friction. 
It is particularly notable that expert-level students have a bidirectional connection between the central idea and 
Newton’s laws of motion. This is reflected in their ability to use these laws to analyze the effects of friction force 
during problem-solving and to subsequently validate the accuracy of their prior reasoning after the problem has 
been resolved. 

In essence, students at the expert level have cultivated an advanced and cohesive cognitive framework, which 
empowers them to adeptly utilize pertinent knowledge to address problems across diverse contexts. 
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Discussion 


By employing quantitative and qualitative methods, this study empirically verifies the differences in knowledge 
structures among students with varying levels of conceptual understanding. These findings are consistent with previ- 
ous research (Bao & Fritchman, 2021; Dai et al., 2019; Linn, 2006; Zou et al., 2023). Furthermore, compared to previous 
studies diagnosing misconceptions about friction force (Ha et al., 2013; Kuzulcuk et al., 2021), this study provides a 
deeper explanation for the origins of these difficulties. The study identifies three levels of students’ understanding of 
friction: novice, intermediate, and expert. By analyzing their reasoning pathways during problem-solving, the origins 
of their misconceptions can be dissected. 

Novice Level: At this level, students’ knowledge structures are fragmented. When addressing problems, their 
primary focus is on the contextual elements, and they depend on their recollection of equations to facilitate the 
matching process. Although they can solve some single-link typical questions using equations or procedural opera- 
tions from memory, they often encounter difficulties with multi-link typical and integrated-link atypical questions. 
The students’ understanding of friction is confined to their everyday experiences, leading them to select answers 
based on experience rather than solving problems through analytical reasoning. For example, they often simplisti- 
cally regard friction force as a resistive force (Caldas & Saltiel, 1995; Clement, 1993), and directly apply the equation 
i = MF, to determine its magnitude (Arons, 1997; Boudreaux & Elby, 2020; Clement, 1993). 

Intermediate Level: Intermediate-level students begin to grasp the passivity of friction force and recognize the 
importance of analyzing the effect of friction force based on the objects’ motion state, rather than merely defining it 
as a resistive force like novice students. However, because the explanatory method of Newton's laws of motion is not 
mastered, they cannot adequately establish a connection between force and motion. They can judge the motion of an 
object by the force on it, and vice versa (Boudreaux & Elby, 2020; Ha et al., 2013). Consequently, these students have 
failed to integrate Newton's laws of motion into their knowledge structure under the topic of friction force. For example, 
in an unfamiliar problem situation, they cannot determine whether there is a tendency for objects to move relative 
to each other (Ha et al., 2013). Students at this level continue to employ memorization strategies to solve problems, 
excelling at addressing questions in familiar contexts but often at a loss when confronted with unfamiliar contexts. 

Expert Level: Expert-level students integrate Newton's laws of motion into their knowledge structure on the topic 
of friction force, thereby establishing a bridge between force and motion, and thus gaining a deeper understanding of 
the passivity inherent in friction. They are capable of solving almost all problems and consistently provide reasonable 
explanations. These students do not rely on memorized equations and are able to apply the central idea consistently 
to solve problems, regardless of whether the context is familiar or unfamiliar (Arons, 1997; Ha et al., 2013). 

Additionally, by comparing the scientific reasoning processes of students with different levels of conceptual 
understanding while solving problems, the progression from superficial to deep understanding is revealed: from the 
accumulation of surface-level contextual information to the construction of more complex knowledge structures, 
and finally to fully integrated conceptual understanding (Bao et al., 2002; Biggs & Collis, 1982). This study found that 
problem situations play a crucial role in this process. Familiar problem contexts can provide a “scaffolding” for students 
to acquire new knowledge, guiding them to start from their pre-existing understanding and integrate it with scientific 
concepts. This is what Piaget (1976) described as the “assimilation” process, which represents the “quantitative change” 
phase in the learning of knowledge. Conversely, unfamiliar problem contexts can offer further opportunities and 
points of leverage for the integration of knowledge, encouraging students to reflect on their personal experiences 
and fragmented knowledge from a scientific and systematic perspective and to modify and reorganize their existing 
knowledge structures. This is the “accommodation” (Piaget, 1976) process, signifying the “qualitative change” phase in 
knowledge acquisition. Hence, the intentional arrangement and meticulous design of problem contexts are essential 
for students to both acquire new knowledge and integrate existing knowledge. The design of unfamiliar contexts 
aims to enhance the integration and deep comprehension of students’ knowledge, discouraging over-reliance on 
familiar contexts and the use of rote learning strategies that lead to superficial learning. 

Moreover, the results indicate that the central idea plays a pivotal role in the integration of knowledge within 
a specific thematic context. Through analyzing the students’ performance in the tests and interviews, we have dis- 
cerned that within the theme of friction force, the passivity of friction plays a pivotal role, reflects its characteristics 
and serves as an anchor to connect other related concepts. Our research findings indicate that the majority of stu- 
dents have not developed a profound understanding of friction force, but rely on extensive problem-solving practice 
to become proficient at answering questions in familiar contexts. However, when faced with unfamiliar scenarios, 
they lack the capability to solve problems. Only a minority of students have a deep understanding of the passivity 
of friction, thereby forming a highly integrated knowledge structure that enables them to flexibly address a variety 
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of problems. Therefore, teachers should tailor their instruction according to the characteristics of students at differ- 
ent levels of knowledge integration. For instance, with novice-level students, teachers should help them overcome 
misconceptions about friction force, such as the misconception that friction force is a resistive force. Teachers can 
provide scenarios where friction force acts as a driving force to induce cognitive conflict, thereby facilitating con- 
ceptual change. For intermediate-level students, instruction should assist in the integration of the central idea with 
Newton’s laws of motion. To achieve this, teachers should help students understand the use of Newtonian mechanics 
explanatory methods (Ha et al., 2013). 


Conclusions and Implications 


This study has constructed a conceptual framework for friction force, assessing students’ levels of understand- 
ing from the perspective of knowledge integration. The research indicates that students’ comprehension of friction 
force can be categorized into three levels: novice, intermediate, and expert. Students at different levels exhibit 
varied reasoning paths when solving problems related to friction force, which also elucidates the reasons behind 
the emergence of various misconceptions during the learning process of friction force. Furthermore, the passivity of 
friction force is identified as a pivotal element in forming an integrated knowledge structure, which is essential for 
achieving profound conceptual understanding. The study provides valuable insights for educators in the development 
of targeted curricula, aimed at enhancing students’ deep comprehension. The findings suggest that constructing a 
conceptual framework is effective for modeling the knowledge structure of students at different levels of conceptual 
understanding. Utilizing this framework, educators can identify gaps in students’ knowledge structures and imple- 
ment targeted instruction to address the specific needs of students at various stages. Furthermore, the central idea 
plays a pivotal role in the knowledge structure, and it is recommended that educators emphasize its importance 
when designing curricula. For example, when teaching about friction force, the focus should be on understanding 
its passivity. This study also has several limitations that warrant attention. Firstly, the research focused solely on static 
and sliding friction force, excluding rolling friction force, thereby necessitating the development and validation of a 
more comprehensive conceptual framework to fully assess students’ understanding of friction force. Secondly, the 
sample was exclusively composed of students from China, which may restrict the generalizability of the findings. 
Future research should aim to broaden the sample to include a more diverse demographic, in order to substantiate 
the applicability of the conceptual framework across different cultural and educational contexts. 
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Appendix 


Friction Force Test 


For the following questions, you may use a value of g = 10 m/s’ as the approximate value for the acceleration due 
to gravity. 


1. As shown in the figure, an object with a mass of 10 kg is moving to the right on a horizontal surface with 
a kinetic friction coefficient of 0.1, during its motion, it is subjected to a horizontal force F to the left with 
a magnitude of 20 N. Regarding the magnitude and direction of friction force acting on the object, which 
of the following statements is true? 


(A) 10 N, to the right V 
B) 10 N, to the left — 


( 
(D) 20 N, to the left 


2. As shown in the figure, a block of mass 0.8 kg is at rest on an inclined plane with an angle of 30° . Ifa horizontal 
force of 3 N, parallel to the inclined plane, is applied to push the block, and the block remains stationary, what is 
the magnitude of the friction force acting on the block? 


(A) 3.N 


3. As shown in the figure, Xiao Ming is riding an escalator upward, and the escalator is accelerating upward Xiao 
Ming is stationary relative to the escalator. What is the direction of the friction force acting on Xiao Ming? 
A) Opposite to the direction of motion. 


B) Same as the direction of motion. 


( 
(B) 
(C) Horizontally to the left. wan 
(D) Horizontally to the right. 

( 


E) No friction force is acting. 


4. As shown in the figure, a conveyor device is placed horizontally with a constant rotational speed. An object is 
gently placed at position A on the conveyor belt, and when position Cis reached, the object has the same velocity 
as the conveyor belt. During the conveyance, what is the situation regarding the friction force acting on the object? 
(A) The object experiences sliding friction in the AC section, directed horizontally 

to the left. 

(B) The object experiences sliding friction in the AC section, directed horizontally 

to the right. 


(C) The object experiences static friction in the CB section, directed horizontally 


to the left. A B =~ 
(D) The object experiences static friction in the CB section, directed horizontally p, 


to the right. 
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5. As shown in the figure, a horizontal force F of 20 N is applied to the right to pull a wooden board, causing it to 
move to the right. The mass of the board is 10 kg, and the kinetic friction coefficient between the board and the 
table surface is 0.1. What is the magnitude of the friction force acting on the board before it tips over? 


(A) 10N 
(B) 20 N 

(C) 

(D) Decreasing from 10 N to 0 


Decreasing from 20 N to 0 


6. During the two moments of Xiao Ming's run, as depicted in the diagrams, f, and f, denote the friction 
forces acting on Xiao Ming in the first and second moments, respectively. Regarding the direction of f, and 
f,, the accurate statement is: 


(A) f, forward, f, backward 

(B) f, forward, f, forward 

(©) f, backward, f, backward 

(D) f, backward, f, forward 7. 2 


7.As shown in the figure, the object is placed on a horizontal conveyor belt, and fixed, smooth horizontal 
rods A and B are installed on both sides of the object, which restrict the movement of the object only in 
between. The mass of the object is m, the kinetic friction coefficient between the object and the conveyor 
belt is 42, and the object moves at a constant speed v,. The conveyor belt moves to the right, its speed 
can be changed. Regarding the friction force f experienced by the object, which of the following state- 


ments is true? B A 

(A) f=F i 

(B) f = umg _ 
(C) The direction of fis opposite to that of F ° 
(D) The direction of fis horizontally to the left. i i i 


8. As shown in the figure, two blocks A and B are stacked together and move to the right on a rough horizontal 
surface under the action of an external force F. If A and B remain relatively stationary during the motion, which of 
the following statements is true about the friction force f experienced by block B? 


(A) If A and B move together at a constant speed, fis directed to the left. 
(B 
(C 
( 


D) If A and B move together with deceleration, fis directed to the left. 


) If Aand B move together at a constant speed, fis directed to the right. 
) 


If Aand B move together with acceleration, fis directed to the left. 


9. As shown in the figure, ona fixed inclined plane, there are objects A and B, and all contact surfaces are rough. 
Objects A and B are relatively stationary and accelerate down the slope together. Regarding the friction force f 
experienced by object A, which of the following statements is true? 


(A) fis directed upward along the contact surface. A 
(B) fis directed downward along the contact surface. 

(C) There is no friction force between object B and object A. 

(D) The motion trend of object A relative to object B is uncertain. 
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10. As shown in the figure, objects a, b, and c, which are identical, are moving on the ground under the force F. 
Assuming the magnitude of F is the same in all three cases. Regarding the friction force experienced by the object, 
which of the following statements is true? 


(A) All three are equal in magnitude 
(B) Object A experiences the greatest mee Seo 
(C) Object B experiences the greatest / 
(D) Object C experiences the greatest a b C 


11. As shown in the figure, Xiao Ming is writing on a horizontal desktop. To prevent slipping, an object is placed 
near the edge of the white paper on the left side. When writing to the right, both the paperweight and the paper 
remain stationary relative to the desktop. Which of the following statements is true? 


(A) The object exerts a friction force on the paper, directed to the left. 


(B) The object exerts a friction force on the paper, directed to the right. desktop 
(C) The desktop exerts a friction force on the paper, directed to the left. object a — | 
(D) The pen exerts a friction force on the paper, directed to the left. Paper 


12. As shown in the figure, a worker suspends a steel pipe between the ceiling and the horizontal ground using 
a lightweight rope. When the pipe is stationary, the rope is vertical. Which of the following statements is true? 


(A) The ground exerts a friction force on the pipe, directed to the right. 


) The ground exerts a friction force on the pipe, directed to the left. 7 


(B 
(C) There is no friction force between the ground and the pipe. 
(D) The motion trend of the pipe relative to the ground is uncertain. 5 


FIVISTTITTITTTI ATTA ITT 


13. As shown in the figure, block P is stationary on a horizontal plane. When two mutually perpendicular horizontal 
forces, F, and F, (F, > F,) , are applied to it individually, the block moves in the direction of F, and F., with the 
respective frictional forces being f, and f.. If both of these horizontal forces are applied at the same time from an 
initial state of rest, the magnitude of the sliding friction force acting on the block is f,. The accurate statement about 
the magnitudes of these three friction forces is: 


A f=h<h 7 J 


@h=h=h, no 
Of>h=f, [vr ; 
0 <= fh; f * ¥4 


14. As shown in the figure, ™, = 2kg , Mm, = lkg , AandB havea kinetic friction coefficient of 0.5, and the horizon- 
tal surface is smooth. When a 10 N horizontal force F is applied to pull A, what is the friction force between A and B? 


(A)ON 


(B) 5 N 
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Abstract. Teachers’ discourse is 
instrumental in facilitating the emergence 
of students’ scientific discourse. Many 
studies have shown that teachers’ cognitive 
demand levels and discursive moves are the 
main factors in eliciting students’ scientific 
discourse, but few focus on whole-class 
(non-grouped) teaching settings. This 
research explored the correlation between 
teachers’ discourse strategies (the cognitive 
demand of questions and discursive 

moves) and students’ scientific discourse 
quality in whole-class teaching without 
intervention. Applying the chi-square test, 
correspondence analysis, and content 
analysis, the study revealed the correlation 
structure between teachers’ discourse 
strategies and students’ scientific discourse 
quality. The teachers’ lower (remembering 
and understanding) cognitive demand 
questions, primarily elicited students’ low- 
quality (everyday discourse and elementary) 
scientific discourse responses. High 
(analyzing and creating) cognitive demand 
questions, primarily elicited students’ high- 
quality (moderate and advanced) scientific 
discourse responses. The teachers’ discursive 
moves also elicited students’ high-quality 
scientific discourse, particularly following 
the teacher's high cognitive demand 
questions. High-control discursive moves 
(e.g., “providing opinion”) elicited students’ 
moderate scientific discourse responses, and 
low-control discursive moves (“allocation 
and organization”) elicited students’ 
moderate and advanced scientific discourse 
responses. Based on these findings, 
suggestions were made for science teachers. 
Keywords: scientific discourse, whole-class 
teaching, teacher's question, cognitive 
demand, discursive move 
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Introduction 


Since the concept of scientific literacy was introduced, it has garnered 
widespread attention from researchers and educators, significantly impact- 
ing science education globally. Currently, scientific literacy is one of the main 
goals of science education, improving the scientific literacy of citizens is the 
fundamental purpose of science education (Norris & Phillips, 2003; Yacoubian, 
2018). Scientific literacy refers to the essence of scientific exploration, being 
able to use scientific concepts in daily life, and being able to read and interpret 
scientific information in mass media (Gillies & Baffour, 2017; Wallace, 2004). 
Rennie considered that people with scientific literacy include those who: 1) 
are interested in and understand the world around them; 2) engage in the 
discourse of and about science; 3) are able to identify questions, investigate, 
and draw evidence-based conclusions; 4) are skeptical and questioning of 
claims made by others about scientific matters; 5) make informed decisions 
about the environment and their own health and well-being (Rennie, 2005, 
p. 11). Among these abilities, scientific discourse plays a vital role (Norris & 
Phillips, 2003), and even scientific literacy can be understood as a language 
ability. Therefore, to cultivate students’ scientific literacy, teachers need to 
promote the use of scientific discourse among students and help them ef- 
fectively participate in scientific discussions. 

Existing research has suggested that the most effective way to develop 
children’s scientific discourse is practice-based teaching, but the effectiveness 
of this way still depends on the quality of classroom dialogue (Gillies & Khan, 
2009; Soysal, 2021b). Teachers’ discourse is a major factor influencing students’ 
scientific discourse. Researchers have explored the relationship between dif- 
ferent types of teachers’ discourse and the production of students’ scientific 
discourse, but often focus on the context of teaching interventions such as 
group inquiry and cooperative learning (Gillies & Khan, 2009; Lee & Irving, 
2018). Few studies focused on naturalistic classrooms (Soysal, 2021a), and 
even fewer considered the teaching realities of large class sizes in East Asia 
and Europe. The limitations of existing research indicate the lack of suitable 
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guidance and measures to help teachers in large classroom settings improve their discourse. To resolve this issue, 
this study explored the correlation between teachers’ discourse and students’ scientific discourse in a whole-class 
teaching setting without grouping. This aimed to identify the parts of the teachers’ discourse strategies that have 
promoted meaning and to help teachers improve their classroom discourse. 


Scientific Discourse and the Level of Students’ Scientific Discourse 


Discourse is a medium for people to participate in social life and a tool for constructing social life. As a fusion 
of socially acknowledged methods of using language, ways of conceptualizing, and modes of communication, 
discourse can be utilized to validate one’s membership in a socially significant group (Gee, 2004). Dewey’s view on 
learning illustrates the important role of discourse, “Learning, in the proper sense, is not learning things, but the 
meanings of things, and this process involves the use of signs, or language in this generic sense” (Dewey, 2018, p. 
164). Knowledge is constructed through the interaction of discourse and other symbolic media within the social 
context. Discourse functions both as a medium for social interaction and as a vehicle for knowledge transmission. 
In essence, to learn a subject is to master the discourse of that subject’s community (Norris & Phillips, 2003; Wal- 
lace, 2004). 

In his work Talking Science: Language, Learning, and Values, Lemke posited, “Learning science means learning to 
talk science. It also means learning to use this specialized conceptual language in reading and writing, in reasoning 
and problem-solving, and in guiding practical action in the laboratory and daily life” (Lemke, 1990, p. 1). From this 
perspective, scientific dialogue differs from conventional social interactions. It emphasizes the shared construction 
of knowledge and comprehension, necessitating the ceaseless negotiation of meanings to cultivate reciprocity 
and mutuality (Mercer, 2010). Furthermore, scientific dialogue employs scientific discourse to construct meanings 
consistently. Scientific discourse entails the processes or methods of discussing scientific information, theories, or 
practices, often involving explanation, reasoning, modeling, evaluation, and so on (Grimes et al., 2018; Lee & Irving, 
2018). Hence, students’ discourse in the classroom should embody two aspects, dialogue with others and integration 
with scientific content. Students need to contribute to the collective process of constructing scientific meanings. 

In an early study, students’ discourse in the science classroom was categorized as detailed explanation, ques- 
tioning, succinct responses, engagement, guidance, and interruption (Gillies, 2004). This categorization primarily 
analyzed students’ discursive moves without adequately integrating scientific content. Similarly, another research 
devised an analytical framework for students’ discourse during discussions, categorizing it into thought and reason- 
ing, complete thought, word/phrase, and no response (Pimentel & McNeill, 2013). Subsequently, a framework to 
assess the scientific level of students’ discourse was introduced, comprising the following levels: social language; 
basic statements; basic use of scientific language; moderate use of scientific language; and advanced scientific 
language (Gillies et al., 2015). Students’ discourse in science classrooms represents a continuum from silence to 
engaging in dialogue in a scientific manner. This framework can systematically encapsulate varying levels of stu- 
dents’ scientific discourse in the science classroom. 


The Facilitative Role of Teachers’ Discourse 


Literature has indicated that involving students in real practice activities benefits the development of their 
scientific discourse. In group collaborative work, students’ social discourse and scientific discourse have increased 
(Grimes et al., 2018; Kulatunga et al., 2013). However, previous research found that cooperative learning promotes 
students’ discourse production; the level of students’ complex thinking like problem-solving and reasoning still 
depends on specific teachers’ discourse strategies (Gillies & Khan, 2009). Without continuous and effective interac- 
tion, students tend not to elaborate on information spontaneously, pose thought-provoking questions, or utilize 
prior knowledge (King, 2002). Similarly, researchers found that coordinating theory and evidence, hypotheses, and 
problems is challenging for students unless supportive tools or guidance are provided (Herrenkohl et al., 2011; 
Kuhn, 1993). The key to developing students’ scientific discourse lies in the specific discourse support of teachers 
in teaching interactions. Therefore, teachers need to be aware of the differences and roles of their discourse to 
support students’ effective participation in scientific dialogue (Bansal, 2018; Lee & Irving, 2018). 

The literature has pointed out that the IRE/F discourse structure of teachers’ initiation, students’ response, 
and teachers’ evaluation or feedback is a typical classroom dialogue structure and still dominates (Geelan, 2013). 
Classroom discourse is predominantly driven by teachers and initiated through their act of posing questions. A 
significant number of researchers have mentioned that the cognitive demands of teachers’ questions are among 
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the factors affecting students’ discourse (Chin, 2006; Smart & Marshall, 2013; Tagnin et al., 2021). According to their 
cognitive demand level, questions can be divided into two categories: low-demand questions and high-demand 
questions (Tagnin et al., 2021). Low-demand cognitive questions prompt descriptions of events, scientific terms, 
and statements, and invoke cognitive processes like memorizing, understanding, and application, which do not 
change the pre-existing knowledge structure. Conversely, high-demand cognitive questions require students to 
engage in higher-level cognitive activities such as analysis, creation, and evaluation, leading to the construction 
of viewpoints and plans. Researchers have found that high-demand cognitive questions can promote students’ 
high-demand thinking and dialogic participation (Soysal, 2021a; Tagnin et al., 2021). The literature also mentioned 
that when teachers’ questions had a low (e.g., understanding) cognitive demand, students’ arguments were at the 
highest level (Soysal, 2021a). 

In addition to questioning strategies, teachers can still use other discourse strategies to promote students 
higher-level scientific discourse. When teachers effectively use the E/F position in the IRE/F structure, they create 
more dialogue possibilities and enhance the complexity of students’ discourse (Wells, 1993). Researchers referred 
to the teachers’ follow-up actions after students’ responses as discursive moves, which might include forms such 
as feedback, evaluation, requests for detailed explanations, or non-verbal behaviors (Bansal, 2018; O'Connor & 
Michaels, 2015; O'Connor & Michaels, 1996). Furthermore, teachers’ discursive moves may involve monologic and 
dialogic instructional purposes (Soysal, 2021¢, p. 230). The former is that teachers provide information, opinions, or 
knowledge for public discourse space (Soysal, 2021b), while the latter is that teachers promote students’ discourse 
interaction and intelligent contribution to classroom conversations (Soysal, 2021c). A series of studies have found 
different teacher discursive moves and elucidated their positive role in promoting classroom dialogue, such as 
“revoicing’, “saying more” (O’Connor & Michaels, 1996), and “marking” (McKeown & Beck, 2004). So, generally, the 
discursive move is a kind of teacher intervention behavior that can promote student participation in classroom 
dialogue (Mortimer, 2003). As an effective tool, teachers can use and need to learn how to use discursive moves to 
continuously support and organize classroom micro-interactions (O'Connor & Michaels, 2015). 


1 


Purpose and Research Questions 


Previous research has indicated that the cognitive demands of teachers’ questions and discursive moves are 
key factors affecting students’ discourse in teaching interactions. It is still unclear about the correlation between 
the cognitive demand of teachers’ questions, teachers’ discursive moves, and the level of students’ scientific dis- 
course. The existing studies have mainly focused on ideal teaching situations created by researchers, such as group 
collaboration and inquiry teaching; however, research focusing on whole-class teaching in large-scale class is still 
limited. Therefore, this study made up for the shortcomings of the existing research and explored the correlation 
between teachers’ discourse strategies (cognitive demand of teachers’ questioning and teachers’ discursive moves) 
and students’ scientific discourse quality under the background of whole-class teaching in large-scale class. Ad- 
ditionally, it provided guidance for teachers to improve their discourse in teaching interaction. Specifically, this 
study answered the following research questions: 

(1) What is the correlation between the cognitive demands of teachers’ questioning and discursive moves 
with the quality of students’ scientific discourse? 
(2) Which types of teachers’ discourse strategies can trigger students’ advanced scientific discourse? 


Research Methodology 
Background 


This study explored the correlation between the cognitive demand of teachers’ questioning, discursive 
moves, and the quality of students’ scientific discourse in whole-class (non-grouped) teaching involving large-scale 
classes. Considering the representativeness of the sample, suitable participants were selected. Beginning in early 
September 2023, with the informed consent of the participants, the researcher entered the science classrooms 
for observation and collected teaching videos, which lasted for nearly four months. During the research process, 
the researcher did not provide any support to the teacher, and the teacher independently conducted classroom 
teaching. The teacher's discourse strategies and students’ scientific discourse quality encoding in this study were 
qualitative categorical variables, which were suitable for correlation judgment using the chi-square test and cor- 
respondence analysis and explained through content analysis. Overall, this was a study conducted in a naturalistic 
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manner, and the teacher's discourse strategies that can trigger the production of advanced scientific discourse in 
students are all derived from the teaching habits of the teacher. 


Participants 


The participants in this study included a lower-secondary school science teacher from Zhejiang Province, 
China, who has four years of experience in science teaching, along with 50 students from her class, representing 
the total enrollment of that class. In China’s current basic education system, large classes are prevalent. Addition- 
ally, due to the pressure of exams, most teachers primarily employ lecture-based teaching methods. To explore the 
correlation between teachers’ discourse strategies and the quality of students’ scientific discourse in the context of 
whole-class teaching and large-scale class, purposive sampling was used in the selection process of participants in 
this study. Firstly, Zhejiang Province is the earliest province in China to adopt and still implement a comprehensive 
science curriculum, which has been in place for over twenty years. Compared to other regions where lecture-based 
teaching remains dominant in China, science teaching in Zhejiang is currently undergoing reforms. Secondly, the 
school is at a moderate level in the city, enrolling students for specific streets and districts, which means that its 
students can, to some extent, represent the general academic level of middle school students in Zhejiang Province. 
Thirdly, the teacher had obtained a master’s degree in subject teaching with a specialization in science education. 
Compared to teachers who had not participated in the comprehensive science teacher training program, this 
teacher was more likely to promote students’ participation in classroom discourse through her daily teaching 
practices. In addition, the researcher established a long-term relationship with the teacher, which facilitated the 
development of the research process. 


The Level Division for Students’ Scientific Discourse 
Students’ scientific discourse analysis framework was employed as the initial coding tool (Gillies et al., 2015). 
During the iterative coding process, this framework was adjusted based on the study’s data to better adapt to the 


analysis of the quality of students’ scientific discourse in whole-class teaching (see Table 1). 


Table 1 
Framework for Analyzing Students’ Scientific Discourse Quality 


Non-scientific discourse Description 
Level 1 silence Refers to the situation where, after a teacher's question, silence ensues for 2 seconds or more before the 
teacher continues speaking. This is interpreted as a student's silent response in the conversational turn. 
Level 2 negative responses Refers to minimal engagement responses, such as simple “yes” or “no” answers or parroting the teacher’s 
words without additional opinion. 
Level 3 everyday discourse Refers to the usage of everyday information and knowledge to answer or pose questions. For example, “It’s 
round”. 
Scientific discourse Description 
Level 4 elementary scientific Refers to the description of phenomena that do not take a propositional form; accurate presentation or appli- 
discourse cation of previously learned scientific knowledge in brief responses; posing open-ended questions about the 


topic to seek explanations or clarifications; presenting ideas supported by reasons. For example, “The edges 
of two adjacent continents fit together, and there are no gaps in between’. 


Level 5 moderate scientific Proposes propositions based on the teaching theme content; makes basic evaluations of representations; 

discourse provides evidence or explanations and clarifications based on scientific knowledge; proposes strategies to 
verify information; participates in ongoing communication, clarifying or supplementing own views or the views 
of others. For example, “What he means to say that the Sun is here, the Earth is next to it, and then the 
Moon moves to this position. It's the Moon, not the Earth, that blocks the light, and this cannot prove that the 
Earth is round’. 


Level 6 advanced scientific Speculates with evidence; presents claim and uses reasoning to establish connections between claim and 

language evidence; provides reasons to question or support others’ claim. For example, “But, if it were to be orbiting 
around the Earth, its size couldn’t be much larger than the Moon’s. At that time, the (Moon’s) gravity (on it) 
wouldn't be stronger than the Earth’s (gravity on it). It would only be a sixth, nowhere near enough to hold it. 
It would have taken off somewhere you wouldn't know’. 
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The Types of Teachers’ Discourse Strategies 


Teachers play a crucial role in pedagogical dialogue, as they initiate and drive the conversation forward by 
posing follow-up questions or employing other discursive moves. To thoroughly explore the correlation between 
teachers’ discourse strategies and the quality of students’ scientific discourse, this study analyzed teachers’ discourse 
strategies including teacher questioning and discursive moves. The revised Bloom's taxonomy was used to analyze 
the cognitive demand level of teachers’ questions (Soysal, 2021a). Regarding teachers’ discursive moves, the previ- 
ous research findings were adopted as a preliminary analytical framework (McKeown & Beck, 2004; O'Connor & 
Michaels, 1996). Through iterative coding, an analytical tool tailored to our dataset for examining teachers’ discourse 
strategies was developed (see Table 2). 

It's important to emphasize that the distinction between various discursive moves lies in the extent of teach- 
ers’ involvement in the classroom dialogue. For instance, “provide opinion (T9)" refers to teachers’ using subjective 
statements or analyses to respond to students, which often stimulate further discussion. “Positive emotion (T8)” 
refers to teachers’ acknowledging students’ effective contributions to the classroom dialogue, indicating teachers 
retain the right to evaluate. “Allocation and organization (T11)” is a type of discursive move in which teachers do 
not intervene in thematic discussions. The main purpose of this move is to promote student participation in the 
classroom and foster the continuity of viewpoints among students. This move highlights the role of the teacher as 
the “facilitator” of classroom dialogue, responsible for arranging and organizing classroom interactions. 


Table 2 
Framework for Analyzing Teachers’ Discourse Strategies 


Question cognitive 


demand Description 


T1 remember Retrieves relevant knowledge or information from long-term memory. For example, “Where does the Earth rotate 


around”. 


T2 understand Determines the meaning of instructional information, including verbal, written, and graphic communication, requires 
students to interpret, summarize, classify, and compare the information that has been presented. For example, “So, 


what is the function of these latitude and longitude lines?” 


T3 apply Executes or uses a procedure in a given situation. 

T4 analyze Breaks down materials into their constituent parts and detects the relationships among the parts and between the 
parts and the overall structure or purpose. For example, “Are there any points in this fact that make you question?” 

T5 evaluate Makes judgments based on standards and norms, such as determining whether evidence is valid, whether a plan 
can verify a problem, and whether a viewpoint is credible. For example, “Which categories do you think are more 
appropriate, more in-depth, and able to grasp some of the essence?” 

T6 create Combines different information or elements to form a novel, coherent whole or create an original product. For 


Discursive move 


T7 action command 


example, constructing an experimental plan based on an inquiry problem, or proposing a viewpoint by synthesizing 
existing information. For example, “What comes to mind when you want to explore this problem?” 


Description 


Requires students to make a behavioral response, such as conducting observations or operations. 


T8 positive emotion 


Directly provides positive emotional feedback, encouraging the student to further supplement; the teacher indicates 
that he/she did not understand, which indirectly provides positive feedback. For example, “Um, good, very good.” 


T9 provide opinion 


Participates in dialogue, expressing personal opinions rather than factual knowledge on the topic of conversation. 
For example, “You represent possibility, but all possibilities must be scrutinized by facts, and must undergo theoreti- 
cal scrutiny.” 


T10 prompt 


Based on the question, provide half of the answer or content that is closely related to the answer, as a hint for stu- 
dents. For example, “Every month, it happens. The moon waxes and wanes, as the ancients have long discovered. 
Our moon waxes and wanes, but who caused this waning?” 


T11 allocation and organiza- 
tion 
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Revoicing expresses an implicit communicative meaning by repeating the student's discourse, such as, “Do you 
think there must be a lot of organisms in places like the Earth’s crust?” 

Establishes a relationship for the speaker based on the presented discourse, such as, “She is standing on his side 
again’; shifts the subject of the discourse towards others, seeking their understanding, such as, “Do you understand 
what she means?” 
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Data Collection and Analysis 


With the informed consent of the participants, the researcher entered the science classroom as an observer 
and collected teaching videos, which lasted for nearly 4 months. To better answer the research questions, the 
following standards for video selection were applied: 1) Students’ discourse entries account for more than a third 
of the total; 2) The videos exhibit a diversity of student discourse, including instances of moderate and advanced 
scientific discourse. Then, according to the transcription standards in Table 3 (Wieselmann et al., 2021), three class- 
room teaching videos were transcribed, resulting in 788 utterances between teacher and students. The teacher's 
utterances accounted for 52% of the total. Based on the research questions, preliminary analysis and screening 
were conducted on the dataset. Dialogues not related to science content, such as class affairs, were removed, as 
well as monologues by the teacher. A total of 446 classroom utterances were coded as categorical variables for 
this study (223 each for the teacher's and students’ utterances). Cohen's Kappa, a robust statistic for measuring 
inter-rater reliability for categorical data, indicated good reliability between the coders (k= .828). All disagreements 
were resolved through discussion. 


Table 3 
Transcription Key 
Symbol Meaning Example 
() Parentheses indicate silence; timed in seconds or untimed indicated by a period in parentheses Um (.) Okay. 
() Parentheses with text indicate a supplement to oral language to make it more understandable If (the land) were flat, 


Teacher-student utterances in this study were coded as qualitative variables. The chi-square test can gener- 
ally indicate whether there is a significant difference between two variables, but further analysis is still needed to 
accurately determine the correlation between the categories of the two variables. Correspondence analysis can 
solve this problem. Correspondence analysis is a qualitative-dominated cross-analysis technique that normalizes 
contingency tables to obtain corresponding matrices, which are then subjected to singular value decomposition 
(de Nooy, 2003; Kutscher & Howard, 2022). Through these processes, the correlation between qualitative data 
is expressed in a reduced dimensional visual space map. The larger the singular value in a dimension, the more 
data variability is explained. Generally, the first two dimensions contain most of the inertia (variation) in the data. 
Greenacre provided a comprehensive description of the concept and mathematical foundation of this method 
(Greenacre, 2017). In summary, correspondence analysis can reduce the dimensionality and visualize the correla- 
tion between high-dimensional categorical variables in data. By observing the angles between vectors and the 
length of the vectors’ projection in the corresponding map, researchers can determine the similarity or correla- 
tion between the two categorical variables. It should be noted that the correlation between variables obtained 
through position and distance in the correspondence analysis map is always determined compared to other points 
(Greenacre, 2017, p. 272). 

This study utilized the Statistical Package for Social Science (SPSS) version 27.0 for data statistical process- 
ing. Firstly, a chi-square test was used to determine whether there was a significant difference between teachers’ 
discourse strategies and students’ scientific discourse quality. Then, correspondence analysis and content analysis 
were employed to explore the specific correlations between teachers’ discourse strategies and the quality of stu- 
dents’ scientific discourse. 


Research Results 
“Remembering” Cognitive Demand Questions Predominated Teachers’ Discourse Strategies 
As shown in Table 4, the majority of the teacher's questions were at the remembering level of cognitive demand 
(23.8%), followed by the understanding level (13.9%). The teacher's discursive moves of “allocation and organization 


(T11)” account for approximately 13.9%, with “action command (T7)" slightly lower at around 13.5%. This suggests 
that the teacher's questions were primarily focused on students’ understanding and memorization of knowledge 
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and information. Additionally, the teacher used various follow-up moves such as “positive emotion’, “prompt”, and 
“allocation and organization” to promote students’ classroom discourse participation. These teacher's discursive 
moves make up 42.6% of the total teacher discourse. However, the effectiveness of these actions was not entirely 
satisfactory, as confirmed by data on the distribution of students’ scientific discourse. 

The statistical result shows that with the overall distribution of teacher’s discourse strategies, elementary 
scientific discourse was the most common response of students (Table 5). Students’ elementary scientific discourse 
is the most frequent (35.9%), followed by everyday discourse (29.1%). Silence and negative responses account for 
approximately 11.2% of the total student discourse, while moderate and advanced scientific discourse accounts for 
about 23.7%. This shows that students could participate in the dialogue, but they mainly used everyday discourse 
and elementary scientific discourse. 


Table 4 
Distribution of Teacher's Discourse Strategies 


Giiestion Remember Understand Apply Analyze Evaluate Create 
(T1) (T2) (T3) (T4) (T5) (T6) 
Frequency 0, 0, 0, 0, 9 9 
(Percentage) 53(23.8%) 31(13.9%) 14(6.3%) 3(1.3%) 17(7.6%) 10(4.5%) 
Action gs . Provide Allocation and 
P ? Positive emotion ra Prompt par 
Discursive move command (T8) opinion (T10) organization 
(17) (T9) (T11) 
Frequency i j é 0 . 
(Percentage) 30 (13.5%) 9 (4.0%) 13(5.8%) 12(5.4%) 31(13.9%) 
Note: The number of teacher's discourse codes is n = 223. 
Table 5 
Distribution of Students’ Science Discourse Levels 
Student Silence Negative Everyday Elementary Moderate Advance 
discourse responses discourse scientific scientific scientific 
Frequency 0, 9 9 0, 0, 9, 
(Percentage) 7 (3.1%) 18 (8.1%) 65 (29.1%) 80 (35.9%) 34 (15.2%) 19 (8.5%) 


Note: The total number of students’ discourse codes is n = 223. 


Teachers’ Discourse Strategies Showed Significant Differences in the Quality of Students’ Scientific Discourse 


A chi-square test was performed to test the difference between teachers’ discourse strategies and students’ 
scientific discourse quality. The results indicated a significant difference between the teacher's discourse strategies 
and students’ scientific discourse quality (p < .01), allowing for the rejection of the null hypothesis. Additionally, the 
strength of the correlation was highlighted by the Contingency Coefficient (C = .626), indicating a strong correla- 
tion between the teacher's discourse strategies (cognitive demand levels of the teacher's questions and discursive 
moves) and students’ scientific discourse quality. 


The Correlation between Teachers’ Discourse Strategies and Students’ Scientific Discourse Quality 


The correspondence analysis was employed to examine the correlation between various categories of the 
two variables: the teacher's discourse strategies and students’ scientific discourse level. Given the research ques- 
tion, discourse codes unrelated to student's scientific discourse were excluded, involving “silence” and “negative 
response”. Then, the corresponding analysis was conducted using the teacher’s discourse strategies as row points 
and students’ scientific discourse levels as column points. 

Table 6 shows the statistical results of the corresponding analysis (x? = 113.208, p < .001), which indicates a 
significant correlation between the two category variables, making it appropriate to proceed with the correspon- 
dence analysis. As shown in Table 6, the total inertia is 0.572. The inertia of Dimension 1 is 0.377 (65.9%), and the 
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inertia of Dimension 2 is 0.158 (27.6%). The two dimensions’ inertia amounted to 0.535, accounting for 93.5% of the 
total inertia. This suggests that the two-dimensional correspondence map explains 93.5% of the data’s variation, 
with minimal information loss. Moreover, as shown in Table 7 and Table 8, the contribution of two dimensions to 
the inertia of each point is greater than .50, all points are well represented in the first two dimensions. Essentially, 
the first two dimensions map can adequately illustrate the specific correlation between the teacher's discourse 
strategies and students’ scientific discourse quality. 


Table 6 
Summary of Correspondence Analysis Statistical Results 


Dimension SV I Prop. of | ¢ p 
1 614 0.377 659 
2 397 0.158 276 
3 192 0.037 .065 
Total 0.572 1 113.208 < .001 


Note: SV = Singular Value; | = Inertia; Prop. of | = Proportion of Inertia; df = 30. 


Table 7 
Contribution of Two Dimensions to Inertia of Row Point 


Row Point 11 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 
Contribution 942 981 .760 996 .620 .997 593 997 .966 .987 .999 
Table 8 


Contribution of Two Dimensions to Inertia of Column Point 


Column Point Everyday Elementary Moderate Advanced 


Contribution 189 116 .992 988 


Observing the corresponding analysis map (Figure 1), it can be found that the points representing students’ 
scientific discourse level are distributed on both sides of the centerline of the vertical axis. Points “Everyday” and 
“Elementary” are located on the left-hand side, while “Moderate” and “Advanced” are located on the right-hand 
side. This indicates that the scientific discourse level of students was divided into two groups with significant dif- 
ferences. On the left-hand side, the teacher's discourse strategies T1 (remember), T2 (understand), T3 (apply), T7 
(action command), and T10 (prompt) are displayed, along with the student scientific discourse levels “everyday” 
and “elementary” On the right-hand side, the teacher’s discourse strategies T4 (analyze), T5 (evaluate), T6 (create), 
T8 (positive emotion), T9 (provide opinion), and T11 (allocation and organize) are displayed, accompanied by the 
student's scientific discourse levels “moderate” and “advanced”. This means that overall, teacher posing high (ana- 
lyze, evaluate, and create) cognitive demand questions and using specific discursive moves (except the “action 
command” move) tended to elicit advanced scientific discourse from students. 

However, the correlation between teacher's discourse strategies and students’ scientific discourse levels 
required further exploration through analyzing the corresponding analysis map. According to the research of 
Greenacre (2017), the strength of the correlation between row and column points can be concluded by the scalar 
product of the vector pointing from the origin to the row and column points. The longer the projection length of 
a row vector on a column vector, the stronger the correlation, while the shorter the projection length, the weaker 
the correlation (Greenacre, 2017, p. 98). Suppose the projection of the row vector on the column vector is located 
on the reverse extension line of the column vector (the angle between the two vectors is greater than 90 degrees). 
It means a negative correlation between the two categories represented by row and column points. Based on 
this, the vectors from the origin to column points (“Elementary’, “Moderate”, and “Advanced”) were employed as 
a reference. The strength of the correlation between different categories of teacher discourse strategies and the 
level of students’ scientific discourse was identified based on the length of their projection onto the column vector. 
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1. Questions with Low Cognitive Demands Elicit Elementary Scientific Discourse from Students 


To further analyze which teacher discourse strategies were closely correlated with students’ elementary sci- 
entific language, the vector “Elementary” was employed as a reference. The projection direction and the length of 
teacher discourse strategy vectors onto this “Elementary” vector were then observed. As shown in Figure 1 (b), the 
projection positions of vectors T1, 12, T3, and T7 are in the positive direction. The projection lengths in descending 
order are vectors T3, T1, T7, T2. This means that teacher questioning at the cognitive demands level of applying, 
remembering, and understanding, triggered students’ elementary scientific discourse. Compared to other teachers’ 
discourse strategies, there was the strongest correlation between the teachers’ questioning of applying cogni- 
tive demand level and the students’ elementary scientific discourse responses. The projections of T5 and T10 are 
located on the reverse extension line of this vector, which means they were negatively correlated with students’ 
elementary scientific discourse. 

Content analysis of teacher-student dialogues also supported this finding. The students’ responses were ob- 
served as everyday discourse or simple academic vocabulary when the teacher’s questions were at low (remember, 
understand, and apply) levels of cognitive demand. In the excerpted dialogue segment in Table 9, the teacher 
posed three questions at the remembering level of cognitive demand (01, 09, 11), which received responses from 
students in everyday discourse (02, 10) and elementary scientific discourse (12). The teacher asked students about 
“the changes in their field of vision during the process of climbing high places” (01) or “the method should be used 
when real experiments cannot be conducted” (11). These questions elicited students to draw upon their existing 
experience or knowledge. Although questioning with low cognitive demand provided opportunities for students 
to participate in classroom conversations, this type of classroom interaction mainly did not trigger students’ high- 
level scientific discourse. 


2. Questions with Creating and Analyzing Cognitive Demands Elicit Students’ Moderate and Advanced Scientific 
Discourse 


To further analyze which teacher discourse strategies had a strong correlation with students’advanced scientific 
discourse, the vector “Moderate” was employed as a reference. The projection direction and the length of teacher 
discourse strategy vectors onto the “Moderate” vector were then observed. As shown in Figure 1(b), the projections 
of teacher discourse strategy vectors on the “Moderate” vector are in the positive direction. In descending order, 
the projection lengths are vectors T6, T8, T9, and 111. This means these teacher discourse strategies were positively 
correlated with students’ moderate scientific discourse responses. Compared to these teacher discourse strate- 
gies, teacher questioning at the creating(T6) cognitive demand level strongly correlated with students’ moderate 
scientific discourse. The teacher’s discursive moves “positive emotion (T8)’, “provide opinion (T9)", and “allocation 
and organization (T11)”, also triggered students’ moderate scientific discourse responses. 
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Using the vector “Advanced” as a reference, the projection of teacher discourse strategy vectors onto vector 
“Advanced’ are in the positive direction and the projection lengths in descending order are vectors T4,T11,T8. The 
remaining T10, T5, and T9 are shorter and closer to each other in length. This means that these teachers’ discourse 
strategies were positively correlated with students’ advanced scientific discourse. In addition, compared to these 
teachers’ discourse strategies, teacher questioning at the analyzing (T4) cognitive demand level had the strongest 
correlation with students’ advanced scientific discourse. The teacher's discursive moves “allocation and organiza- 
tion (T11)” and “positive emotion (T8)” followed. The “prompt” move, “provide opinion” move, and questions at 
the cognitive demand level of evaluating also elicited students’ advanced scientific discourse responses, but their 
correlations with students’ advanced scientific discourse were weakest. 

Content analysis of teacher-student dialogues also supports these findings. The students’ responses were 
observed at high levels (moderate and advanced) of scientific discourse when the teacher used high (analyze and 
create) cognitive demand questions. As shown in Table 9, the teacher's question at the creating level of cognitive 
demand (07) hoped that students construct a research plan to explore whether “To see a thousand miles, ascend 
another floor” can explain that the Earth is a sphere. A student proposed a hypothesis using everyday discourse 
(08), which was a restatement of the information already presented in the previous conversation. To publicize the 
information “use simulation experiments’, the teacher initiated two questions at the remembering level of cogni- 
tive demand (09, 11). Following this, the teacher restated the initial question (07) at the creating level of cognitive 
demand (13), which triggered the student's response using moderate scientific discourse (14). Immediately following, 
the teacher triggered the student to propose a plan (16) with “allocation and organization” (15), and the teacher 
gave “positive emotion” feedback (17), which elicited the student to clarify and supplement the proposed plan (18). 


Table 9 
Classroom Dialogue Segment 


Turn 
Nuniber Excerpt Code 
01 T: There’s a saying that goes, “If you wish to see farther, climb to a higher level.” So, whathappens —-T1 remember 
to the range of what you can see when you go higher? 
02 S: The broader. L3 everyday discourse 
03 T: Do you think this statement has any relation to the shape of the Earth? T5 evaluate 
04 (2) L1 silence 
05 T: If the Earth were flat, like the ancient saying “the sky is round and the Earth is square,” would T9 provide opinion 
the higher you go mean you could see a broader range? Or is it only when the Earth is a sphere, a 
curved surface, that the higher you go, the further you can see? 
06 (3) L1 silence 
07 T: What comes to mind when you want to explore this question? T6 create 
08 $1: Um (.) it's just(.) | was thinking that (.) if (the land) were flat, looking out from a tall building, | L3 everyday discourse 
think it might be the same. 
09 T: So, do we have a chance to find a flat Earth to climb up and see? T1 remember 
10 S: Impossible. L3 everyday discourse 
11 T: Um, no, we can't. It’s impossible to explore. So, what can we turn to at this point? T1 remember 
12 S: A simulation experiment. L4 elementary scientific 
13 T: We can also conduct simulation experiments or use models, we can directly use images. So, how _—‘T6 create 
can we use images to explore? 
14 $2: We can draw a straight line on the blackboard now, and then draw a person on it. Then, looking = L5 moderate 
at that person, the point you see might be at a lower place. This is looking down from a high place, 
and then what you see from a low place with this high place (in comparison). 
15 T: Do you understand? T11 allocation and organi- 
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Turn 
Number Excerpt Code 
16 $3: | think it's like assuming you have a range of vision from this eye. It's about whether you can L5 moderate 
see the same object from a lower position as you can see from a higher position. If you can see it 
from both positions, it means that the places you can see from a high position are not visible from a 
low position, which can prove that the Earth is not flat. 
17 T: I'm not quite following, you can draw it. T8 positive emotion 
18 $3: So, this is like a flat surface, right? And there’s one person up high, and another one down low. L5 moderate 


Just imagine they're both in the same building, looking out. If I’m up high and can see certain stuff 
within this area, and | can see the same things when I’m down low, that’s what it would be like if the 
Earth were round. But if the Earth were flat, then being up high in the same building, | could see up 
to this point, but being down low, | could only see up to here. 


3. Similarities Exist Among the Types of Teacher's Discourse in Triggering Student Scientific Discourse 


Interestingly, the analysis of the corresponding map reveals that specific teachers’ discourse strategies shared 
similarities in triggering students’ scientific discourse production. In Figure 1 (b), smaller angles are observed 
between the vectors T1 (remember) and T3 (apply). Similarly, smaller angles are found between the other two 
sets of vectors: T2 (understand) and T7 (action command), T8 (positive emotion) and T9 (provide opinion). Smaller 
angles between the positive vectors of different categories of the same variable suggest a greater similarity be- 
tween these two categories. Among the eleven teacher discourse strategies, these three sets of teacher discourse 
strategies shared similarities in triggering students’ scientific discourse. Specifically, T1 and T3 are closely related 
to “Elementary” which means that their function of eliciting students’ elementary science discourse was similar. 
T2 and T7 are located in the middle of “Everyday” and “Elementary” suggesting that they could trigger students’ 
responses using daily and elementary scientific discourse. T8 and T9 are relatively close to “Moderate”, which could 
trigger students’ moderate scientific discourse responses. As demonstrated in the dialogue segment in Table 9, 
the teacher's discursive moves of “positive emotion” (15) and “provide opinion” (17) both preceded instances of 
students’ moderate scientific discourse (16, 18). 


Figure 2 
The Correlation Structure of Teachers’ Discourse Strategies and the Quality of Students’ Scientific Discourse 
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In summary, the findings indicated a correlation structure existed between teacher's discourse strategies and 
the quality of students’ scientific discourse (Figure 2). Teachers’ questions at the applying level of cognitive demand 
were most likely to elicit students’ elementary scientific discourse. Similarly, questions at the creating level were 
most likely to elicit students’ moderate scientific discourse. Finally, questions at the analyzing level were most likely 
to elicit students’ advanced scientific discourse. 


Discussion 


This study explored the correlation between the teacher's discourse strategies and the quality of students’ 
scientific discourse within the context of large-scale class during whole-class teaching. An analytical framework 
aligning with the dataset was developed, building upon established research. A chi-square test was conducted to 
ascertain whether a significant difference existed between the teacher's questions, the teacher's discursive moves, 
and the quality of students’ scientific discourse. Following this, correspondence analysis and content analysis were 
employed to delve into the specific correlation further. 


The Strong Correlation Between High-Demand Cognitive Questions from 
Teachers’ and Students’ Scientific Discourse 


Consistent with existing research, teachers often use questions that demand “remembering” and “understand- 
ing’, which are of low cognitive demand (Huang et al., 2019; Saka & Inaltekin, 2023; Soysal, 2021a). This study shows 
that teachers’ discourse strategies were observed at the low (remember, understand, and apply) cognitive demand, 
nearly half (43.9%) of the total numbers (Table 4). This implies that the teacher initiating dialogue was mainly to 
check students’ knowledge, not to understand their thoughts. Therefore, many students’ discourse was observed 
as non-scientific (40.3%) and elementary scientific (35.9%). Of course, these findings were obtained in large-class 
size with whole-class (non-grouped) teaching. But, if this non-dialogic authoritative discourse is the only form of 
classroom dialogue, it will be detrimental to students’ meaning construction. 

Fortunately, questions with high (analyze, evaluate, and create) cognitive demand were observed in this study, 
although they only accounted for a small percentage (13.4%) of the total teacher discourse strategies (Table 4). 
Previous research indicates a direct association between high cognitive demand questions and the complexity 
of students’ scientific discourse, with high-level questions provoking more complex responses, while lower-level 
questions eliciting simpler ones (Ni et al., 2014; Soysal, 2021a; Tagnin et al., 2021). The findings of this study cor- 
roborated the results of previous research. Teachers’ high (create, analyze) cognitive demand questions are also 
strongly correlated with students’ high-quality (moderate and advanced) scientific discourse, suggesting that high 
cognitive demand questions are crucial for fostering high-quality scientific discourse responses in large class sizes 
with a whole-class (non-grouped) teaching setting. 


Functions of Teachers’ Discursive Moves 


Numerous teacher's discursive moves were found in the dataset, accounting for 43.6% of the total teacher 
discourse. Notably, the discursive moves of “allocation and organization” appeared frequently, making up 13.19%. 
This demonstrates that the teacher's practice in China is not just seemingly participatory teaching (Huang et al., 
2019), but actively uses discursive moves to facilitate students’ contributions to knowledge production (Figure 2). 
As shown in Table 9 through content analysis, the teacher's discursive moves elicited students’ high-quality scientific 
discourse, particularly following the teacher's questions of high cognitive demand. It is reasonable to infer that 
students struggled when they faced questions of high cognitive demand, at this point, the teacher's subsequent 
discursive moves served as a supportive scaffold (van de Pol et al., 2018). 

This study shows that teacher discursive moves fostering student scientific discourse production included 
“prompt”, “provide opinion’, “positive emotion’, and “allocation and organization”. People interact in discourse to 
construct meaning, negotiate, and explore their identities (Mercer, 2010). Halliday pointed out that discourse has 
the following functions: the conceptual function of conveying information; the interpersonal function of express- 
ing emotions, attitudes, and acting on others; and the cultural function of constructing activities that conform to 
specific communities (Halliday, 1993). These discursive moves observed in this study primarily served cognitive, 
interpersonal, and cultural functions in organizing and coordinating scientific dialogue. “Prompt” and “provide 
opinion” constitute cognitive support provided by the teacher, intended to enable students to construct or re- 
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fine a complex cognitive process by incorporating new perspectives or information. “Positive emotion” offered 
interpersonal affirmation and encouragement, stimulating student participation in the dialogue. “Allocation and 
organization” move denoted the teacher's deliberate withdrawal from the conversation. In this move, the teacher 
assumed the role of a facilitator, rather than a dictator of the discussion. This discursive move indicated that the level 
of teacher control diminished, transitioning from providing viewpoints and support to delegating the responsibility 
of knowledge production to the students. 

This study reveals that teacher discursive moves with higher control (such as “provide opinion”) were closely 
correlated with moderate-level student scientific discourse, and those with lower control (such as “allocation and 
organization”) were closely correlated with both moderate and advanced-level student scientific discourse. Based 
on these findings, in whole-class teaching (not grouped), to enhance students’ production of moderate and ad- 
vanced scientific discourse, teachers should design and use high cognitive demand questions, and flexibly employ 
discursive moves to provide cognitive, emotional, or cultural support to students. However, this study does not 
provide teachers with a strategy selection plan tailored to specific discourse contexts, such as different types of 
student responses. This limitation needs further research, which focuses on the analysis of teacher-student dialogue 
sequences, exploring the selection and outcomes of the teacher's discourse strategies in different dialogue contexts. 


Conclusion and Implications 


Collectively, this research demonstrates a significant correlation between teachers’ discourse strategies 
and the quality of students’ scientific discourse within whole-class teaching. High cognitive demand questions 
posed by teachers can trigger the production of students’ scientific discourse, and the facilitating role of teachers’ 
discursive moves cannot be overlooked. As primary participants in classroom dialogue, science teachers need to 
understand the differences in the quality of students’ scientific discourse and the crucial role of teacher questions 
and discursive moves. To effectively promote students’ scientific discourse production, teachers should employ 
questions of high cognitive demand and flexibly utilize their discursive moves. This study focuses on whole-class 
teaching with large class sizes, which is overlooked by most research. Additionally, this study provides indicators of 
teachers’ discourse strategies and their functions in fostering students’ scientific discourse, which can help science 
teachers understand and reflect on classroom discourse and improve their discourse. 

The scope of this study is confined to a single science classroom, involving one teacher and 50 students, thereby 
limiting its potential for broader generalization. However, through reasonable purposive sampling and linking results 
with existing research, this study showed that these findings do not significantly deviate. Furthermore, as this study 
was conducted in China, there is a clear need for expanded research to further explore the correlation between 
teachers’ discourse and the scientific quality of students’ discourse in whole-class teaching across varied contexts. 
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